Merge dev changes into main. (#15)

* Code clean-up
* Fix issue where one SP event can have multiple file contexts if one doc references another, which would crash the import.
* If copilot context URL is a direct URL to item, we can load that too.
* SPO components GitIgnore
* DevConfig PS Json ignore
* Key vault configure task errors due to differing Az & O365 tenants don't cause install abort, just gives a scary warning.
* PR: Add Copilot events data in the reports (#6)

---------

Co-authored-by: Sam Betts <[email protected]>

.gitignore

@@ -18,6 +18,7 @@
 /src/Misc/packages
 **/.vscode
 *.tmp
+*.tmp.*
 /src/SPO/Scripts.AITrackerInstaller/AITracker/builduntdeploy.ps1
 /src/AnalyticsEngine/Web/Properties/PublishProfiles
 /src/SPO/Scripts.AITrackerInstaller/DevConfig.json
@@ -27,3 +28,4 @@
 
 # Avoid having the pbix files in the repo
 /reports/Usage Analytics/*.pbix
+/src/SPO/AITracker/DevConfig.json

docs/Release Notes.md

@@ -1,4 +1,7 @@
-## master - NEW
+## master - build 1400
+Reliability updates for copilot event processing. Previously, SharePoint metadata lookups failed for file specific actions. Also now misc chats are better imported for Copilot actions in Outlook and other app-hosts. 
+
+## master - build 1375
 Fixes for SharePoint site aggregation usage stats. When looking up a site ID for a previously imported site (with no ID set), the site lookup save would crash. 
 
 No SQL schema changes.

docs/analytics/README.md

@@ -192,3 +192,17 @@ The information is displayed in the matrix that can be changed dynamically based
 7. At this point, we can publish the report to the same workspace as the previous one.
 
 >**Important!** If you want to extend the report to load additional data, create new calculated columns or measures, you should use this report template and not `Analytics_DataModel.pbit`.
+
+## Additional considerations
+
+### Audit events
+
+For audit events to be crawled, the table dbo.OrgUrls must have at least one row. If you are not using SPO Advanced Analytics your table will be empty.
+
+By default, `Audit.SharePoint` and `Audit.General` are crawled.
+
+You can add a new row with any content to enable audit logging crawling:
+
+```SQL
+INSERT [dbo].[org_urls] ([url_base]) VALUES ('this-is-here-so-audit-data-can-be-imported');
+```

docs/analytics/data_model.md

@@ -24,47 +24,82 @@ The date shown in the report will be a Monday, that represents the activity of t
 
 The following metrics show the number of activities of this type performed by the user. These are available in the `ActivitiesWeekly` and `ActivitiesWeeklyColumns` tables.
 
+### Activity metrics
+
 **OneDrive**
-- Shared Externally
-- Shared Internally
-- Synced
-- Viewed/Edited
+* Shared Externally
+* Shared Internally
+* Synced
+* Viewed/Edited
 
 **Outlook**
-- Emails Read 
-- Emails Received
-- Emails Sent 
-- Outlook Meetings Created
-- Outlook Meetings Interacted
+* Emails Read 
+* Emails Received
+* Emails Sent 
+* Outlook Meetings Created
+* Outlook Meetings Interacted
 
 **SharePoint**
-- Shared Externally
-- Shared Internally
-- Synced
-- Viewed/Edited
+* Shared Externally
+* Shared Internally
+* Synced
+* Viewed/Edited
 
 **Teams**
-- Private Chats
-- Calls
-- Meetings Attended
-- Meetings Organized
-- Adhoc Meetings Attended
-- Adhoc Meetings Organized
-- Scheduled OneTime Meetings Attended
-- Scheduled OneTime Meetings Organized
-- Scheduled Recurring Meetings Attended
-- Scheduled Recurring Meetings Organized
-- Audio Duration Seconds
-- Video Duration Seconds
-- Screenshare Duration Seconds
-- Urgent Messages
-- Post Messages
-- Reply Messages
+* Private Chats
+* Calls
+* Meetings Attended
+* Meetings Organized
+* Adhoc Meetings Attended
+* Adhoc Meetings Organized
+* Scheduled OneTime Meetings Attended
+* Scheduled OneTime Meetings Organized
+* Scheduled Recurring Meetings Attended
+* Scheduled Recurring Meetings Organized
+* Audio Duration Seconds
+* Video Duration Seconds
+* Screenshare Duration Seconds
+* Urgent Messages
+* Post Messages
+* Reply Messages
 
 **Yammer**
-- Liked
-- Posted
-- Read
+* Liked
+* Posted
+* Read
+
+**Copilot**
+
+Hosts where Copilot has been used:
+* Assist365
+* Bing
+* BashTool
+* DevUI
+* Excel
+* Loop
+* M365AdminCenter
+* M365App
+* Office
+* OneNote
+* Outlook
+* Planner
+* PowerPoint
+* SharePoint
+* Stream
+* Teams
+* VivaCopilot
+* VivaEngage
+* VivaGoals
+* Whiteboard
+* Word
+
+**Copilot interactions**
+
+* Number of chat interactions.
+* Number of files where there was an interaction (ie. used Copilot to create a slide in PowerPoint).
+* Number of meeting where Copilot was used.
+
+### Usage metrics
 
 The following metrics' values are true/false. These are available in the `UsageWeekly` table.
 
@@ -81,54 +116,61 @@ App per platform
 | <ul><li>Outlook Windows</li><li>Word Windows</li><li>Excel Windows</li><li>Powerpoint Windows</li><li>Onenote Windows</li><li>Teams Windows</li></ul> | <ul><li>Outlook Mac</li><li>Word Mac</li><li>Excel Mac</li><li>Powerpoint Mac</li><li>Onenote Mac</li><li>Teams Mac</li></ul> | <ul><li>Outlook Mobile</li><li>Word Mobile</li><li>Excel Mobile</li><li>Powerpoint Mobile</li><li>Onenote Mobile</li><li>Teams Mobile</li></ul> | <ul><li>Outlook Web</li><li>Word Web</li><li>Excel Web</li><li>Powerpoint Web</li><li>Onenote Web</li><li>Teams Web</li></ul>
 
 **Teams Devices by Platform/OS**
-- Used Web
-- Used Mobile
-- Used Windows
-- Used Linux
-- Used Mac
-- Used Chrome
-- Used iOS
-- Used Android
-- Used WinPhone
+* Used Web
+* Used Mobile
+* Used Windows
+* Used Linux
+* Used Mac
+* Used Chrome
+* Used iOS
+* Used Android
+* Used WinPhone
 
 **Yammer Devices by Platoform/OS**
-- Used Web
-- Used Mobile
-- Used Others
-- Used iPhone
-- Used iPad
-- Used Android
-- Used WinPhone
-- Platform Count
+* Used Web
+* Used Mobile
+* Used Others
+* Used iPhone
+* Used iPad
+* Used Android
+* Used WinPhone
+* Platform Count
 
 ## Raw data
 
 The raw data for the report can be found in the following tables of the database:
 
 ### Activity tables
 
-- [dbo].[onedrive_user_activity_log]
-- [dbo].[outlook_user_activity_log]
-- [dbo].[platform_user_activity_log]
-- [dbo].[sharepoint_user_activity_log]
-- [dbo].[teams_user_activity_log]
-- [dbo].[teams_user_device_usage_log]
-- [dbo].[yammer_device_activity_log]
-- [dbo].[yammer_user_activity_log]
+* dbo.onedrive_user_activity_log
+* dbo.outlook_user_activity_log
+* dbo.platform_user_activity_log
+* dbo.sharepoint_user_activity_log
+* dbo.teams_user_activity_log
+* dbo.teams_user_device_usage_log
+* dbo.yammer_device_activity_log
+* dbo.yammer_user_activity_log
 
 ### User details
 
-- [dbo].[users]
-- [dbo].[user_license_type_lookups]
-- [dbo].[user_company_name]
-- [dbo].[user_country_or_region]
-- [dbo].[user_departments]
-- [dbo].[user_job_titles]
-- [dbo].[user_license_type_lookups]
-- [dbo].[user_office_locations]
-- [dbo].[user_state_or_province]
-- [dbo].[user_usage_locations]
-- [dbo].[provinces]
-- [dbo].[license_types]
-- [dbo].[countries]
-- [dbo].[cities]
+* dbo.users
+* dbo.user_license_type_lookups
+* dbo.user_company_name
+* dbo.user_country_or_region
+* dbo.user_departments
+* dbo.user_job_titles
+* dbo.user_license_type_lookups
+* dbo.user_office_locations
+* dbo.user_state_or_province
+* dbo.user_usage_locations
+* dbo.provinces
+* dbo.license_types
+* dbo.countries
+* dbo.cities
+
+### Audit events
+
+* dbo.audit_events
+* dbo.event_copilot_chats
+* dbo.event_copilot_files
+* dbo.event_copilot_meetings

docs/install.md

@@ -150,7 +150,7 @@ The application should be now created. You need the client ID, secret value, and
 
 The runtime service principal must have permissions granted to the activity and Graph API. This is not something granted by default, so needs to be added after registration.
 
-The list of permissions can be found in the [prerequisites docs](prerequisites.md#prerequisite-permissions).
+The list of permissions can be found in the [prerequisites docs](prerequisites.md#permissions).
 
 In the runtime app registration, add the permissions in the following screenshot. Click "API permissions" and then "Add a permission".