We were having high CPU usage in the authoring server and by looking at logs one of the marketing automation activity which were keep running even though there was no active plan exists currently in the website.
The the custom activity were using the redbus to deliver SMS to xDB contacts and the activity was failing by throwing an exception which in turn put the activity to “redbus” error queue.And the message bus was retrying the activity which were taking lot of CPU cycles.(Rebus use a separate thread pool which is very CPU intensive)
Debugging steps to find the plan
The authoring server throwing below exception
INFO [SendSmsXXXX] deferred message ’86c27c9d-c075-4715-a94e-2f8945802ce5′ to ‘9ed4910d-ab14-2b00-0000-0589aaca4733’
Since this activity was executing by the Marketing Automation Engine , i have take look the logs in MA OP service role and got the below message.
2019-05-01 17:44:46.906 +10:00 [Information] Enrolling contact ‘9ed4910d-ab14-5500-0000-0589a99ff47e’ into plan ‘da545732-35e5-4bff-9d08-5132695d27a0’ with context ‘””‘
2019-05-01 17:44:46.921 +10:00 [Information] Transition enrollment “ActivityEnrollmentKey: { ContactId: 9ed4910d-ab14-5500-0000-0589a99ff47e, PlanDefinitionId: da545732-35e5-4bff-9d08-5132695d27a0, ContextKey: , ActivityId: 3e03292c-5eed-95e6-d180-8e43071b3fae }” to activity ‘d2b92ffc-31ec-3ba1-63e2-c3e18485105f’
2019-05-01 17:44:46.937 +10:00 [Information] SendSmsActivity: ContactId 9ed4910d-ab14-5500-0000-0589a99ff47e, MessageId: 86c27c9d-c075-4715-a94e-2f8945802ce5
2019-05-01 17:44:46.937 +10:00 [Information] Transition enrollment “ActivityEnrollmentKey: { ContactId: 9ed4910d-ab14-5500-0000-0589a99ff47e, PlanDefinitionId: da545732-35e5-4bff-9d08-5132695d27a0, ContextKey: , ActivityId: d2b92ffc-31ec-3ba1-63e2-c3e18485105f }” to activity ‘f35a3db8-5760-ce6f-53ec-a4b7fe5e4f63’
2019-05-01 17:44:46.953 +10:00 [Information] Contact ‘9ed4910d-ab14-5500-0000-0589a99ff47e’ exiting plan ‘da545732-35e5-4bff-9d08-5132695d27a0’ with context ‘””
From the above error message I was able find the plan that was executing the plan ‘da545732-35e5-4bff-9d08-5132695d27a0’ .
I have search for the plan under the Marketing automation dashboard but it was not displayed in the active campaign list.
After a search the plan with the Id was found in the “recycle” bin.
Even Though the plan was deleted from dashboard the definition of the plan was not removed from the reference data database.I have tried to deploy the marketing definition hoping it will overwrite the definition.Unfortunately the definition were not removed from the reference database.
Finally I have to manually remove the definitions from the reference database.
Query to list the automation plans-
select * from [xdb_refdata].[DefinitionCultures] where ID IN ( SELECT b.ID
FROM [xdb_refdata].[Definitions] a, [xdb_refdata].[DefinitionCultures] b where a.ID = b.ID and TypeID = ‘E9C5A97B-7F1A-4276-AB0F-E28A1AE8A762’)
select * from [xdb_refdata].[Definitions] where TypeID = ‘E9C5A97B-7F1A-4276-AB0F-E28A1AE8A762’
Query to remove the automation plans-
delete from [xdb_refdata].[DefinitionCultures] where ID IN ( SELECT b.ID
FROM [xdb_refdata].[Definitions] a, [xdb_refdata].[DefinitionCultures] b where a.ID = b.ID and TypeID = ‘E9C5A97B-7F1A-4276-AB0F-E28A1AE8A762’)
delete from [xdb_refdata].[Definitions] where TypeID = ‘E9C5A97B-7F1A-4276-AB0F-E28A1AE8A762’