Duplicated logs ingested into Sentinel with OCI (Azure Functions) Data Connector

[fa-clavis]

Describe the bug
Hello team,

When I was going to model and create some analytics rules for a client, I noticed that there were multiple logs with the same "id_g" field populated into the OCI_Logs_CL table.

This "id_g" field (in OCI logs it's the eventID) is the unique identifier for which every alert receives, so there shouldn't be multiple logs with the same ID in Sentinel.
Source: https://docs.oracle.com/en-us/iaas/Content/Audit/Reference/logeventreference.htm

I have installed the default Azure Functions with the ARM template, while I have crosschecked multiple eventIDs in the OCI audit logs, I'm not sure why the same log is being populated 2,3,5, 30 times or more.

See the attached screenshots to understand the issue better.

P.S. I suspected something was odd with the connector because the amount of logs that's being ingested because the amount of logs that are produced in OCI is much less than the client's AWS environment.

To Reproduce
Steps to reproduce the behavior:

1. Install the data connector via ARM: Oracle Cloud Infrastructure (using Azure Functions)
2. Create all the required backend logging on the OCI side
3. Insert all the client details in the template and in "CursorType" instead of "group", insert "partition"
4. Wait for the logs to populate into the table
5. In the Sentinel Workspace, query the following table "OCI_Logs_CL" and look for the eventIDs in the "id_g" column

Expected behavior
The expected behavior is to ingest each eventID once and not multiple times.

Screenshots

Additional context

To install and configure the data connector, I used the following resources:

1. https://docs.oracle.com/en/learn/stream-oci-logs-to-azure-sentinel/index.html#related-links
2. https://learn.microsoft.com/en-us/azure/sentinel/data-connectors/oracle-cloud-infrastructure

While the log is unique within the Audit Log service, I strongly believe there is something wrong with the Function App or the Python script where the cursor isn't being set up correctly as it pages through the logs.

I believe the problem is somewhere near this line in the main.py file:

Azure-Sentinel/Solutions/Oracle Cloud Infrastructure/Data Connectors/AzureFunctionOCILogs/main.py

Line 131 in 296f272

def process_events(client: oci.streaming.StreamClient, stream_id, initial_cursor, limit, sentinel: AzureSentinelConnector, start_ts):

[fa-clavis]

Hello team,
Any updates regarding this issue?

[v-sudkharat]

Hey @fa-clavis, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!

[fa-clavis]

Hello @v-sudkharat, any updates regarding this issue?

[v-sudkharat]

Hey @fa-clavis, We are checking the function code for duplicate ingestion of data, just want to know, it's possible to you to share OCI demo account with us? so we can check our changes. Or please let us also know if you have any test workspace into your environment where you can test our change. Thanks!

[fa-clavis]

Hello @v-sudkharat , I talked with the client here, and they don't have a test environment available for OCI, however, they prefer to do the troubleshooting in a quick meet.
Is there any way we can schedule this meeting through e-mail or Teams?

[v-sudkharat]

Hi @fa-clavis, let us check with our team to validate some changes, which required the OCI env. once it gets validated will update you. Thanks!

[fa-clavis]

Hello @v-sudkharat, great news!
Hope it gets fixed ASAP!

[v-sudkharat]

Hi @fa-clavis, Could you please share few events generated with same id_g with us via a mail - [email protected]
to understand the duplication of data in each event.

[v-sudkharat]

Hi @fa-clavis, Just want to check with you, Is there any multiple function app deployed into the environment and point towards the same workspace?

[v-sudkharat]

@fa-clavis, Waiting for your response on above comment, so based on that we can reach out to respective team. Thanks!

[v-sudkharat]

Hi @fa-clavis, Any response for us?

[fa-clavis]

Hello @v-sudkharat, sorry for the delay.
No, there is only 1 function app which is now in the "stopped" status.

[v-sudkharat]

@fa-clavis, thank you for the update, we are connecting with our concern data connector team for this issue, we will keep you updated. Thanks!

[fa-clavis]

Hello @v-sudkharat, any updates regarding this issue?