Processing items with Work Item Timer Jobs in SharePoint 2010
Development, Performance, Productivity, SharePoint, SharePoint Gems
Once in a while you find yourself in a situation where you have to process some items in SharePoint. Using Timer Jobs in only a half of the answer. Find out how to process items using Work Item Timer Jobs in SharePoint 2010.
Processing items in SharePoint 2010
One of the things we have probably all done at least once during our SharePoint developer career was to create a solution for processing a number of items, such as ratings, subscriptions, some kind of requests, etc. We have all learned that such processes can take quite a while to complete and it is therefore a good practice to use Timer Jobs for implementing the logic. Because Timer Jobs are being executed outside the w3wp.exe process, they are not only less prone to failures due to process termination but also allow us to move the load away from the Application Pool serving our solution.
When dealing with items you have to have some kind of queue which describes what should be processed. Most frequently such queue contains some input information for the task (rating, username, etc.) but it may also contain ID’s of sites and items to which the task refers.
While implementing queues for Timer Jobs many people choose Lists for storage. And although there is nothing wrong with this approach, it requires some additional work in creating and maintaining the queue List’s schema and all the plumbing for adding and cleaning queued items not to mention supporting upgrades should anything change in the future! There is however an easier way to work with item queues in SharePoint 2010.
Presenting Work Item Timer Jobs
Work Item Timer Jobs (SPWorkItemJobDefinition) are specialized types of Timer Jobs designed for dealing with item queues. And although they have been around for quite some time (available as a part of Windows SharePoint Services v3) they haven’t been that well documented yet and there are not many samples to find of how they can be used.
Creating Work Items
At the base of every Work Item Timer Job is the Work Item: a unit of work that is picked up and processed by the Job when it executes. A work item can be added using the SPSite.AddWorkItem method, eg.:
Guid siteId = SPContext.Current.Site.ID;
Guid webId = SPContext.Current.Web.ID;
Guid listId = SPContext.Current.ListId;
int itemId = SPContext.Current.ItemId;
Guid itemUniqueId = SPContext.Current.ListItem.UniqueId;
int currentUserId = SPContext.Current.Web.CurrentUser.ID;
SPSecurity.RunWithElevatedPrivileges(() => {
using (SPSite site = new SPSite(siteId)) {
site.AddWorkItem(Guid.NewGuid(),
DateTime.Now.ToUniversalTime(),
MyJobDefinition.WorkItemTypeId,
webId,
listId,
itemId,
true,
itemUniqueId,
itemUniqueId,
currentUserId,
null,
"Hello World",
Guid.Empty);
}
});
Although the number of parameters required by the AddWorkItem method might seem scary it is all way easier than it looks.
First there is the gWorkItemId parameter which uniquely identifies the work item.
The second parameter (schdDateTime) determines when the item should be processed. One important thing to notice here is that the time should be stored in the Universal Time format. Without this you might find yourself creating work items in the past what would prevent the Timer Job from processing them.
The next parameter (gWorkItemType) contains the ID of the work item type. This value is very important as it’s used by the Work Item Timer Job to pick up its work items. The identifier set while creating a Work Item should match the value returned by the SPWorkItemJobDefinition.WorkItemType method.
Next there are a few parameters that can be used for retrieving the List Item to which the Work Item refers to. For example when working with ratings you would have the item that has been rated (List Item) and the rating Work Item which contains the rating value. In order to update the average rating on the List Item you would need a reference to the Site and List where the particular item is stored but also the ID of the item itself so that you can retrieve and update it.
Then there is the nUserId parameter which contains the ID of the user who requested the Work Item. This can be useful for tracking purposes.
Next, there are two parameters which can be used for storing input values for the Timer Job and which replace the need for a whole separate List. Those parameters are rgbBinaryPayload and strTextPayload and can be used to store respectively binary and text payloads. When working with ratings you could for example store the rating value (eg. 4) as the text payload. In other scenarios you could choose to store more complex objects: either serialized as JSON/XML as the text payload or as a binary payload. Which one you use depends mostly on the type of Timer Jobs that you are building.
Important: there are two things that you should keep in mind when designing your solution based on Work Items. First of all the assembly that contains the code which adds a new Work Item must have Full Trust. Secondly the user who adds the Work Item must be a Site Collection Administrator. In most scenarios you will therefore be adding new Work Items after elevating privileges.
Calling the SPSite.AddWorkItem method results in adding a record to the ScheduledWorkItems table in the Content Database of your Site Collection.
Those Work Items records are cleaned up once the item has been processed by the Timer Job.
Now we have our Work Item created let’s create a job that will process our Work Items.
Creating a Work Item Timer Job
As mentioned before, Work Item Timer Job are specialized types of Timer Job. From the deployment perspective they look the same as regular Timer Jobs, so they also need a Feature to be deployed and a schedule to execute. The big difference is in the execution process. Where you would implement the Execute method in a regular Timer Job, you use the ProcessWorkItem method to execute your logic.
The following code snippet presents a sample Work Item Timer Job.
public class MyJobDefinition : SPWorkItemJobDefinition {
public static readonly string WorkItemJobDisplayName = "My Work Item Job";
public static readonly Guid WorkItemTypeId = new Guid("{CEAAFFA4-4391-40D6-868E-19EDEDB78DD4}");
public override Guid WorkItemType() {
return WorkItemTypeId;
}
public override int BatchFetchLimit {
get {
return 50;
}
}
public override string DisplayName {
get {
return WorkItemJobDisplayName;
}
}
public MyJobDefinition() {
}
public MyJobDefinition(string name, SPWebApplication webApplication)
: base(name, webApplication) {
}
protected override bool ProcessWorkItem(SPContentDatabase contentDatabase, SPWorkItemCollection workItems, SPWorkItem workItem, SPJobState jobState) {
if (workItem == null || String.IsNullOrEmpty(workItem.TextPayload)) {
throw new ArgumentNullException("workItem");
}
try {
using (SPSite site = new SPSite(workItem.SiteId)) {
using (SPWeb web = site.OpenWeb(workItem.WebId)) {
try {
SPList list = web.Lists[workItem.ParentId];
SPListItem listItem = list.GetItemByUniqueId(workItem.ItemGuid);
string message = workItem.TextPayload;
listItem["Message"] = message;
listItem.SystemUpdate(false);
}
catch (Exception ex) {
// exception handling
}
finally {
// SubCollection(): required to set the ParentWeb property on the WorkItemsCollection
// Delete(): required to remove the item from the queue
workItems.SubCollection(site, web, 0, (uint)workItems.Count).DeleteWorkItem(workItem.Id);
}
}
}
}
catch (Exception ex) {
// exception handling
}
return true;
}
}
We begin with defining the name for our Timer Job and the Work Item Type. Next, using the BatchFetchLimit property we specify how many items our Timer Job should process in a single run. Depending on the logic of your job this can help you keep your environment from overloading and the job running for too long. Finally, in the ProcessWorkItem method, we implement our logic for processing Work Items. In this example we retrieve the associated List Item and we update the value of its Message field with the message stored as the Text Payload of our Work Item.
As you can see this is all very straightforward and allows you to focus on the real job instead of the plumbing.
Important: One very important thing that you should implement correctly and test thoroughly is cleaning up Work Items after they have been processed. Without this they would be processed every time the Timer Job executes over and over again.
In the code sample above you can see how a processed Work Item is being removed in line 52. Although the ProcessWorkItem method receives the collection of Work Items you cannot just remove the Work Item from it. For the Work Item to be removed the collection has to have a reference to the Parent Web and therefore you have to call the SubCollection method first before calling the DeleteWorkItem method.
Summary
Work Item Timer Jobs are specialized type of Timer Jobs in the SharePoint platform, that have been designed to work with queued items. The mechanics behind the Work Item Timer Jobs contain all the plumbing required for managing queues allowing you to focus on processing items. Work Item Timer Jobs are a great alternative to using custom Lists as they require less maintenance and are a part of the SharePoint platform.
September 16th, 2011 at 2:44 pm
One small suggestion.
"Secondly the user who adds the Work Item must be a Site Collection Administrator. In most scenarios you will therefore be adding new Work Items after elevating privileges."
Would impersonating the system account work? I try to keep people away from RWEP.
- Paul
September 17th, 2011 at 8:47 am
@Paul: The SPSite.AddWorkItem calls the SPWeb.CurrentUser.IsSiteAdmin property so as long as it returns true you're good to go.
September 19th, 2011 at 4:08 am
I always used Windows Services that access Sharepoint API, never been used Timer Job. What do you think ?
Thanks.
September 19th, 2011 at 6:46 am
@itsdone: The problem with Windows Services is that there is lack of deployment mechanism for them. Timer Jobs are a SharePoint-native capability and therefore my primary choice.
September 27th, 2011 at 1:58 pm
Will this work in a "least privileges" configured farm? I.e. one where the application pool account won't have access to the configuration database?
(I'm guessing that as it's a timer job, the answer might be 'no')
September 28th, 2011 at 3:37 pm
@Andrew: Application Pool does not have to have access to the configuration database. The Work Item is being created in the Content Database of the current Site Collection and the Timer Job is running in the owstimer.exe process which should be able to read the Work Items from Content Databases. So I guess it's a 'yes', isn't it?
October 11th, 2011 at 4:27 pm
Yup, it is – I've just used one, in a least privileges system, and it's fine. I'd missed that point – that the work item isn't stored in the Config database. That makes these fantastically useful! Love it!
January 19th, 2012 at 6:38 pm
Waldek, I am hoping you can help me with this. I followed your post all most to the letter. The WorkItem is being added to the database but it never is never being kicked off. I double and triple check that my date is UTC, checked the log files for trace information, attached to OWSTimer process but nothing.
Any thoughts on where I can look to track down the problem?
January 20th, 2012 at 8:20 am
Are you sure that the Timer Job is running?
February 11th, 2012 at 5:24 pm
I have tried your code, it works well, but the work item is not being deleted from the database…what could be wrong?
February 21st, 2012 at 5:41 pm
@Wayne: Have you tried to step through the code with debugger just to be sure that all the data is in place? Additionally, have you included the code from line 52?
October 10th, 2012 at 2:39 pm
I have the same problem as Shaune Donohue, entry appears in [dbo].[ScheduledWorkItems], but is never executed. Is it possible to force execution? I can not find the job in the administration (job status / definition etc), but it is not weird as long as it is not in future activation – correct? I also have a contructor with custom parameters – does it make a difference? Thanks!
October 10th, 2012 at 4:26 pm
Does your constructor call the base constructor? As far as I know your Timer Job should be visible in CA. Are you sure that it has been registered properly and if it's working?
October 30th, 2012 at 1:29 pm
Waldek, I followed your instructions to the letter but for some reason my job is not being called. My assemblies are GAC'ed. Any ideas? Also, do you know which timer job processes those work items? any information can help. Thanks!
October 31st, 2012 at 7:01 pm
Have you tried debugging your Timer Job to see if it's picking up the work items at all?
November 28th, 2012 at 11:16 am
Hello Waldek, I hope you can help me.
We have an event receiver somewhere that adds work items. Then a workitem timer job is supposed to pick up those work items.
I attached the debugger to the ProcessWorkItem method is not fired. THe constructors are being fired, so I am not sure whats wrong
http://sharepoint.stackexchange.com/questions/52567/workitem-timerjob-wont-execute/52570#52570
November 29th, 2012 at 7:16 pm
Have you checked in the database if the work items are being created? Another thing that might prevent the timer job from picking up the work items is either wrong Work Item ID or wrong time, so try checking those as well.
December 3rd, 2012 at 1:55 pm
Thanks for the code. But you did not mention something important. The code alone does not work. I tried it and it put the record into the database table. But there was no timer job to pick it up. I thought there is a time job in SharePoint that goes through the database, pick up the values there and then create and start the appropriate SPWorkItemJob. This seems not to be the case. In another article, the guy also created the Job itself from the SPWorkItemJobDefinition. And this is the problem, because the job cannot being created through the code where you add the work item (the WebApp user cannot add something to the SP config database). Don't you have to create the job and if not, how could sharepoint know which job to create only through the WorkItemType?
December 3rd, 2012 at 4:20 pm
There is a whole section in the article about creating a Timer Job to process the work items. Have you read through it? Is there any information that you are still missing?
December 4th, 2012 at 9:26 am
It is really a nice article on the subject, but what I and, apparently Dieter Geiss missed in this text is a clarification that SPWorkItemJobDefinition is in fact a descendant from SPTimerJob:
"Work Item Timer Job are specialized types of Timer Job. From the deployment perspective they look the same as regular Timer Jobs, so they also need a Feature to be deployed and a schedule to execute."
I think it should be in big bold letters, because otherwise many developers (and I amongst them) would believe that it is sufficient just to write a class with WorkItemId and add a WorkItem in site collection, and then just wait for timer job to start all by itself =)
Also I want to ask a question:
When creating SPWorkItemJobDefinition, one should define its schedule – minute, hour, etc.
For example – my timer job is scheduled to run, say, every hour, And my work item is added with schdDateTime = "next day at 00:00:35"
So, I wonder:
* Will my workitem appear in ProcessWorkItem at the first start of timer job (i.e. before 00:00:35)?
* Will ProcessWorkItem of job definition fire strictly at 00:00:35?
So, should I check in my code that this is not a right time for my workitem to execute?
December 4th, 2012 at 10:33 pm
1) Not necessarily: it depends what else code do you have running in your job prior to processing items
2) Theoretically yes, but it all depends on other process that are executed in parallel, how much time they take and how many items are processed in a single batch. The good news is that SharePoint automatically picks up work items for you that should be picked by the particular job run, based on their scheduled time so you don't have to worry about that
December 12th, 2012 at 1:16 pm
A problem I encountered with this mechanism is that if your job runs for too long, SharePoint will mark it as if it didn't ran (probably some sort of a timeout mechanism), which will make it run in loops forever.
Any clue about that?
Thanks.
December 14th, 2012 at 9:16 am
Not really, no. Although it might be worth investigating if one of the base classes would allow you to define the timeout period.
January 11th, 2013 at 10:07 am
Thanks for the anwer. I know how to create a timer job. And I know how to do that. I just thought that I don't have to start it BEFORE I would put items in the database. That means my timer job has to be installed during deployment of the whole solution and will be present all the time in the timer jos list. It cannot created only once on the fly (a run once job) when SharePoint picks up my items. This would have been perfect becuase in my case the time job would run very seldomly.
January 11th, 2013 at 10:10 am
Thank you very much for the anwer. I know how to create a timer job. And I know how to do that. I just thought that I don\'t have to start it BEFORE I would put items in the database. That means my timer job has to be installed during deployment of the whole solution and will be present all the time in the timer jos list. It cannot created only once on the fly (a run once job) when SharePoint picks up my items. This would have been perfect becuase in my case the time job would run very seldomly.
January 11th, 2013 at 10:13 am
You could do that theoretically. The only thing you should take into account would be the number of work items stored in the database. If it would be more than the amount configured in your Timer Job, you would have to run it multiple times in order to process all items. The challenging part is that the only way to discover how many work items there are is by browsing the database which is far from perfect and very likely not something you would be able to do on a production system.
On the other hand if you would choose a scheduled execution, if there were no Work Items, the job would just finish quickly upon starting having very little impact on the rest of the system.
January 23rd, 2013 at 5:32 pm
Not even theoretically. The problem is that if the user is initiating the operation where a timer job is needed and the timer job is not already there, it won't be possible to create it because the user does not have rights to create timer job not even a run once job. Using ElevatedPrivileges in the code also won't help because the web app user can normally not access the SharePoint Config DB. This would have been the perfect solution: The user clicks a button (like importing a bunch of data). The code creates a run-once timer job to do the work. The work is through and the timer job is gone.
January 25th, 2013 at 2:04 pm
The challenge with this approach is recreating the job every time you need it to run which introduces way more overhead comparing to having a job without a schedule present and setting the run-once scheduled when necessary.
April 24th, 2013 at 4:50 pm
Hi Waldek,
Thanks for this blog post! Very handy :-)
A question: A client of mine has a lot of document sets (5000+ docsets per library, 11 libraries). If a field on a document set is updated, the shared fields of the child documents are synced via the DocumentSetEventReceiver.
If a docset contains more than 10 items, the DocumentSetEventReceiver adds a workitem to the content database, to sync the fields via the 'Document Set fields synchronization job'.
However, about a year ago, someone decided to remove the event receiver hooks from the doclibs because they produced a lot of errors. Now, the client wants the sync functionality back so I re-hooked the event receivers.
I saw that the functionality isn't working and when I checked the ScheduledWorkItems table, I saw that it is flooded with tasks (2000+), almost all tasks contain a document set title in the 'textpayload' column. Testing shows me that these are typical DocumentSetEventReceiver jobs.
So, I want to remove these jobs to clear the scheduled table, hopefully it will work again then. Simply running the Document Set fields synchronization job doesn't work, maybe because the InternalState column of all records is '9'.
I know it's not allowed to alter the database records, so I tried to get a reference to the old workitems. I tried something like SPWorkItemCollection workItems = new SPWorkItemCollection(site, new Guid("FC4C2610-25F1-4FE8-8E90-D132E16C2D91"));, however, I only get a handle on the jobs that are recently added (about 1 or 2).
Do you have any idea how I can get a handle on all jobs in a proper way, so I can clear the scheduled table?
Thanks!
April 24th, 2013 at 6:11 pm
Have you tried using the SPWorkItemCollection.SubCollection method (http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spworkitemcollection.subcollection.aspx) instead to retrieve the old work items?
April 24th, 2013 at 6:14 pm
It's a little bit awkward, but it works, using reflection:
(site is your SPSite, guid is your workitem guid)
SPWorkItemCollection items = new SPWorkItemCollection(site, guid);
uint rows, columns;
object workerItemsObject;
items.GetType().GetProperty("GetAllWorkItems", BindingFlags.NonPublic | BindingFlags.Instance).GetSetMethod(true).Invoke(items, new object[] { true });
site.GetWorkItems(items, out columns, out rows, out workerItemsObject);
Cheers.
April 24th, 2013 at 6:19 pm
Sorry, when I wrote workitem guid I meant work item TYPE guid.