Having been exposed to Kubernetes and Microsoft's Service Fabric, the following are some of my notes about both platforms:

Similarities#

They are both orchestrators and pretty much can handle:

• Container Images Hosting
• Scaling
• Healing
• Monitoring
• Rolling Updates
• Service Location

However, Service Fabric support for containers came recently. Initially, Azure Service Fabric was mostly an orchestrator for .NET processes running Windows only.

Strengths#

Kubernetes:#

In 2017, k8s became an industry standard. Every Cloud vendor offers full support and some offer Kubernetes as a Service such as Azure Kubernetes Service (AKS) where the vendor takes care of the cluster creation, maintenance and management.

Given this, it became obvious that if any business is planning to adopt Microservices as a developmemt strategy, they are most likely thinking about employing Kubernetes. Managed services offered by many Cloud vendors such as Azure AKS makes this decison a lot easier as dvelopers no longer have to worry about provisioning or maintaining k8s clusters.

Besides the huge developer and industry support that it is currently receiving, K8s is a joy to work with. Deployments can be described in yaml or json and thrown at the cluster so it can make sure that the desired state is realized. Please refer to this post for more information.

Not sure about these:

• Can k8s create singletons?
• Can I have multiple depoyments of the same Docker instance with different enviroment variables?

Service Fabric:#

In my opinion, one of the most differentiating factor for Service Fabric is its developer-friendly programming model. It supports reliable stateless, stateful and actor models in a powerful yet abstracted way which makes programming in Service Fabric safe and easy. Please refer to earlier posts here and here for more information.

In addition, Service Fabric supports different ways to host code:

• Guest Executable i.e. unmodified Win32 applications or services
• Guest Windows and Linux containers
• Reliable stateless and stateful services in .NET C#, F# and Java
• Reliable actors in .NET C#, F# and Java

The concept of app in Service Fabric is quite sophisticated allowing developers to create an app type and instantiate many copies of it making the concept work well in multi-tenant deployments.

Weaknesses#

Kubernetes:#

I am not in a position to state any weaknesses for k8s. But, from (perhaps) a very naive perspective, I would say that:

• The lack of standard programming models is definitely something that can be improved.
• K8s can only work with containers! So any piece of code that must deployed to k8s must be containerized first.
• Currently k8s is only a Linux orchestrator. Although a beta 1.9 version is said to support Windows containers.

Service Fabric#

One of the major setbacks for Service Fabric (or at least the public version that was made available) is that it was conceived at a time when k8s is burgeoning into an industry standard. It is becoming very difficult for Microsoft to convince developers and businesses to adopt this semi-proprieytary platform when they can use k8s.

There are also a couple of things that can be improved:

• Service Fabric relies on XML documents to describe services and configuration.
• Reliance on Visual Studio although it is possible to do things in Service Fabric without Visual Studio as demonstrated here

Future#

I love Service Fabric! But unfortunately (I say unfortyantely because I actually did spend a lot of time on it), I don't think it has a particularly great future given the strength and the momentum of k8s. Ideally Microsoft should continue to suppprt k8s in a strong way, perhaps port its Service Fabric programming model to work on k8s using .NET Core and eventually phase out Service Fabric.

I wanted to assess Azure Cosmos DB to see possibilities of converting some of our backend applications to use this database technology to provide a globally distributed database that we desperately need as our services now cover a bigger geographic location. However, this post is not about Cosmos DB specifically! It is mainly about notes about the architecture of the pieces that surround Cosmos and how I am thinking to implement them.

Macro Architecture#

This is how I imagines our system to look like:

In this post, I would like to concentrate only on the components that are boxed in orange. Namely, the Web API layer and the two processors.

Azure App Service, Docker & Kubernetes#

We have been using Azure App Service for a while and we are happy with it. We have experience in managing it, scaling it and configuring it. I did not want to change that. So I decided to continue to use Azure App Service to host our Web API layer which will be written in .NET Core. This layer communicates with Cosmos DB to provide a multi-regional access to our customers.

I wanted to monitor Cosmos DB changes to launch different Microservices in the form of Azure Functions, Logic Apps or other processes. I could use Azure Functions to track the Cosmos DB changes but I decided to write my own little .NET Core stand-alone Console app using the Microsoft Change Feed library which makes things quite easy.

Normally, I use WebJobs to handle queue processing and I do have a lot of experience with this. However, in .NET Core, the deployment of a WebJob is not very clear to me so I decided to write a stand-alone console app based on WebJobs SDK but can be deployed somewhere else.

To host and deploy the two stand-alone .NET core console apps i.e. Change Feeder and Queue Processor, opted to make them Docker images and deploy them to a Kubernetes cluster.

Containerizing the WebJobs Console App#

I based my code on this blog post by Matt Roberts. My solution has several projects...but two are important for this step: AvalonWebJobs and AvalonWebDal. AvalonWebDal is a class library that has common functionality that I depend on. I used the following Docker file to build the WebJobs console app:

I used the following Docker command to build the image:

and the following to test locally:

Containerizing the Change Feeder App#

I based my code on this documentation post by Microsoft. My solution has several projects...but two are important for this step: AvalonChangeFeeder and AvalonWebDal. AvalonWebDal is a class library that has common functionality that I depend on. I used the following Docker file to build the WebJobs console app:

I used the following Docker command to build the image:

and the following to test locally:

Important note:

Make sure that the .csproj project file contains the following item groups so that the appsettings will be available in the container. Failure to do so will cause an error message unable to find appsettngs.json and it is not optional:

Kubernetes#

I used this documentation link to spawn a test 1-node k8s cluster in Azure. The process is really simple and quick. I tagged and published my two container images to an Azure Container Registry using this documentaion link.

Now time to actually deploy to the k8s cluster.

Deployments#

Becasue I wanted to scale the two web jobs and the change feeder separately, I opted to create two deployments: one for the web jobs and another for the change feeder. Alternatively, I could have used a two-pod deployment but this will have meant that my two containers will need to be scaled the same since the unit of scaling in k8s is the deployment ...not the pod. Please refer to this stack overflow issue for more information.

I used the following .yml file for the WebJobs k8s deployment:

and the following kubectl command to deploy:

I used the following .yml file for the Change feeder k8s deployment:

and the following kubectl command to deploy:

Observations#

Initialy I set the number of replicas to one so I can make sure that everything is running well before I scale the k8s deployments.

After the deployment as above, I used kubectl get pods to see the status of the nodes. I noticed that the webjobs pod is in running state (which is desired) while the change feeder container is in CrashLoopBackOff state. Humm....after some reasearch, I found this helpful stack overflow issue. So I used the following command to see the actual console logs:

After fiddling with it for a while, I discovered the problem has to do with how the change feeder console app was coded. It uses a typical Console.ReadKey method to make the application run until a key is pressed. So I modified my code based on this useful code snippet, rebuild my container and re-deployed and yes....the change feeder pod is in running state.

It took me a while to get the above to work. This is because I was updating the image in the container registry without changing the tag. This did not force a re-pull and I ended up not using the latest image that I was deploying to the container. It seems that k8s caches the images unless you add to the deployment file a imagePullPolicy: Always in the container spec. Doing so forces k8s to re-pull the image. The other option is to change the image tag.

Now things look better ...but there is another problem. Upon termination, the change feeder container needs to de-register and clean up some resources. Unfortunately I noticed when I issue a docker stop <id>, the process is abruplty terminated and there is no change for the change feeder thread to clean up. I found a good article that describes how to gracefully stop Docker containers which goes into some detail to describe the best way to handle it. However, since I am using a .NET Core app, I really did not find an easy way to handle the two Linux signales: SIGINT and SIGTERM. I did find a lot of discussions about it here. As it stands now, if I run the container in an intercative mode using docker run --rm -ti avalonchangefeeder, for example, and then perform control-c or control-break, the container shuts down gracefully. However, if I issue a docker stop, the container abruplty exists without giving any chance for the change feeder to do any cleanup :-(

Application Insights#

I used Application Insights to trace and track what is happening inside the container. This provide a really powerful way to monitor what is happening inside the container. I noticed that the Application Insights has a Kubernetes extension....but I am not using it yet.

Scaling#

Now that everything is in Kubernetes, we can scale the deployments up and down as we need. Scaling the web jobs deployment up to multiple replicas provide several consumers of the queue and scaling the change feeder deployment up to multiple replicas automatically adjusts the divide the work among themselves.

This is just a note about Azure CLI login options.

Option 1:#

Login interactively via a browser

Option 2:#

The best way to login is to use an Azure Service Principal though. So I registered an application in Azure Directory i.e. AzureCliScriptApp and assigned a service principal. I will use this service principal to login.

Create a service principal:#

Make sure you are in the same tenant that you want to authenticate against. If not, use 'az account set --subscription "your-subs"' to set the account.

To display the Azure Directory apps:

The above will yield the app id ...a big string that looks like this: e68ab97f-cff2-4b50-83d5-eec9fe266ccc

To login with service principal:

Useful Commands:#

List all subscriptions

Set the default account

List the Clouds

Kubernetes#

if you are using the Azure CLI to provision a Kubernetes cluster, you should use this command if you used the service principal to login

Where: $rgName is the PowerShell variable that holds the resource group name $k8sClusterName is the PowerShell variable that holds the k8s cluster name $spAppId is the PowerShell variable that holds the service principal app id $spPassword is the PowerShell variable that holds the service principal password $k8sNodesCount is the PowerShell variable that holds the k8s cluster desired nodes count

Refer to this doc for more information

I started with this documentation page to learn about Azure durable Functions. I wanted to know if I can build a way to implement actors in Azure Functions. Actors Programming Model is pretty interesting and I did some work on it before.

Following the Azure Functions sample instructions mentioned in the above link, I quickly got up and running. However, I wanted to answer the following questions about actors in Azure Functions:

• Create a new actor giving a provided actor id
• Signal an existing actor to perform something
• When do actors get created?
• When do actors get terminated?
• Can we read the actor's internal state?
• What about .NET Core and .NET Standard 2.0 and other stuff?

Create a new actor:#

I created an HTTP trigger that looks like this where I provide a code that can be used as an instance id for the singleton i.e. membership actor. If the membership actor status is null or not running, then I start it with a StartNewAsync:

Signal an existing actor to perform something#

If the membership actor does exist, we raise a refresh event to wake up the singleton so it can do work:

The actual membership actor code looks like this:

Multiple signals#

But what happens if the actor is signaled frantically via raising an external event from an HTTP trigger, for example? The event signals are actually enqueued to the instance so they should run as many times as they are sginaled.

If you are observing the actor's streaming logs when you try this, it could get very confusing. This is because durable functions manage long-term running processes in short-lived functions is by taking advantage of state retrieved in the context and replaying the function to resume at the next step (from this article). Effectively what you will see if that functions are started, completed and re-started again to resume state.

Code Delays#

Singletons should not use Task functions such as Task.Delay(millis) to simulate code delays. This will cause run-time errors:

The preferred way for delays or timeouts is:

Where deadline is defined:

It is very important that we leverage the context to provide accurate timer information as opposed to TimeSpan and DateTime.Now, etc. I have seen very varying (not correct) results when I used TimeSpan.FromMinutes(30), for example.

Wait on multiple events#

What if we want the actor to wait on an external event or on a internal timeout event to perhaps refresh our membership periodically? I created another membership function i.e. E3_MembershipWithTimer that awaits on either an operation event or a timeout event:

When do actors get created?#

Actors or singletons actually do persist in storage (please see the section about termination)......this is how an Azure Functions knows how to start them when it restarts. So if you create actors with specific instance ids (or actor ids), shut down the functions and restart it, the singleton instances are available. When you want to trigger an instance, you must check its running state and then invoke the proper API:

When do actors get terminated?#

They can be easily terminated using the TerminateAsync API. So I created a little HTTP trugger that would terminate instances:

The Azure Durable Functions maintain a state of all running instances in a task hub which is basically a storage resource with control queues, qork-item queues, a history table and lease blobs. You can read more about this here.

Effectively, the host.json durableTask indicate the hub name:

The run-time environment stores related information about running instances in storage keyed by the hub name.

The actor state#

Each actor has an internal state! It is initially read by the singleton as an input:

and it is updated using:

But it seems that the internal state is actually not persisted anywhere ...it is transient. When actors are initially created, a state is passed as an input i.e. context.GetInput<dynamic>() and the actor updates it with a call to ContinueAsNew which actually restarts itself with a new state.

The internal state can be read by using one of the APIs of the instance management:

Where client is DurableOrchestrationClient. The status input is the actor's internal state:

I am not sure if the actor internal state is meant to hold big state though. Perhaps it is better if the actor exposes its state externally so HTTP triggers, for example, can read it directly from the external store.

One way of doing this is to modify the code to look something like this:

and the membership actor:

and the HTTP trigger that retrieves the membership actor state from an extenal source without dealing with the actor:

So unlike regular actor implementation, Azure Functions singletons do not expose any method to be called from the outside! The platform only allows starting/creating, querying and terminating instances.

Comments & Anamolies#

.NET Core and .NET Standard 2.0#

It is work in progress! It is best to use the .NET full framework with Azure Durable Functions. Hopefully this will change soon and we will be able to use .NET Core reliably.

Local Debugging#

I had a very hard time with this. The symptoms that I experienced are unfortutanely not experienced by other developers who tried this as I could not see similar reported issues. I am using Vs2017 and Azure Functions Extension ...the latest at the time of writing DEC/2017.

Here are my comments:

• If you want to debug locally, make sure you set both the local.setting.json and host.json file to copy always. You do this from the properties window.
• On both of my developer machines, I hit F5, it prompts me to install the Azure Functions Core tools and things look quite good. I was able to run locally.
• But then subsequent F5, I get very different results ranging from:
  • The CLI starts and exits on its own ...I could not find out what the reason is
  • The CLI starts and displays the functions URls. But it also complains about some files were changed and the host needs to restart. The URls are not responsive and there is nothing you can do except to terminate and restart.
  • The CLI statrs and actually works.....does not happen often ...but I have seen it work
  • F5 mostly causes the CLI to start and exit. Control-F5 does not exit ...but the function URLs are not accessible due to this change detected message.
• Effectively, local debugging did not work for me at all. It was a very frustrating experience. So I had to deploy everything (see deplyment below) to Azure and debug there....another frustrating experience.

Deployment#

• The only effective way I was able to find out how to deploy a Durable Functions App is via Visual Studio. I have heard some people got it to work with VSTS. But, given that this is a test run, I did not really explore this option.
• However, if you just click the publish button in VS, it will auto-create a storage account for you which names things really weird. My recommendation is to create the App Service, Service Plan, Storage and App Insights in portal or via Azure CLI and then use Visual Studio to publish into it.
• If you renamed a function and re-deployed, Visual Studio will publish the new functions app with the new function. But the old function will still be there (you can see it from the portal). You can then use Kudo, navigate to the directory and actually delete the old function folder.
• The local.settings.json entries are not deployed to Azure! This means you have to manually create them in the portal app settings or in Visual Studio deployment window.

Storage#

As mentioned, an Azure Storage is required to maintain teh durable instances. They are keyed off the hub name you specify in the host. There are entries in blob, tables, files and queues.

Logging#

Unless you turn on streaming on a function in the portal, you don't get anything (or at least, I could not find a way to do it). But watching the log from the portal is very difficult as it times out and it is not very user friendly. This is one area that requires better UX in the portal. The logs are also stored on the app's file system which you can access from Kudo. However, I noticed that, unless you request stream logging on a function, these files are not created.

So the story of logging is a little frustrating at best! I had to use App Insights trace to see what is going on.

Timers#

As mentioned above, it is very important that we leverage the context to provide accurate timer information as opposed to TimeSpan and DateTime.Now, etc. Initially I used TimeSpan.FromMinutes(30) to wait for 30 minutes....but the way to do it is to always use the context such as DateTime deadline = context.CurrentUtcDateTime.AddMinutes(30);. After doing that, I started getting conistent timeout periods and things worked normally.

Instance Termination#

Although TerminateAsync on an instance works, I am not exactly sure if it works the way it is supposed to:

• If I have a running instance and that instance is actually waiting on an external or time out event, TerminateAsync does not do anything. I guess because a message is enqueued to the instance but the instance is waiting on other events ....so it did not get the terminate signal yet.
• If the instance is not waiting on anything, TerminateAsync replays the instance which runs code that you don't necessarily want to run. For example, I had an instance that triggers a logic app once it receives an end operation which works. However, if I terminate the instance using TerminateAync, the code triggers the logic app again because it was replayed!

Not sure if this behavior is correct and what the terminate signal actually do.

Conclusion#

Reflecting on the little time that I spent with Azure Durable Functions, I think they will play an important part of my future solutions. I think they have what it takes to use as actors especially that the Azure Durable Functions are designed to support 1000's of instances . If we externalize the actor's state, we will be able to query the external store as opposed to query the actors themselves to retrieve their state.

Azure Durable Actors can also employ reminders and other sophisticated techniques found in Service Fabric actors such as long-running, stateful, single-threaded, location-transparent and globally addressable (taken from the overview documentation page). However, as stated above and unlike other actor implementations, Azure Functions singletons do not expose methods that can be called from the outside.

In any case, the Azure Durable Functions are still in preview. So we can expect many great features to be added soon.

This PowerShell script creates self-signed root and client certificates, export them and import what is needed:

I hope it helps someone.

I saw many posts about deleting documents in Azure DocumentDB...but none of them worked quite well for me. So I spent a few hours on this and finally got it to work. Below is my solution. The following posts helped me tremendously (thank you):

• https://talkingaboutdata.wordpress.com/2015/08/24/deleting-multiple-documents-from-azure-documentdb/
• https://www.tutorialspoint.com/documentdb/documentdb_delete_document.htm
• http://stackoverflow.com/questions/29137708/how-to-delete-all-the-documents-in-documentdb-through-c-sharp-code
• https://azure.microsoft.com/en-us/blog/working-with-dates-in-azure-documentdb-4/

I basically wanted to delete aging documents (based on number of hours) from a collection. So my final routine looks like this. Below is some explanation:

Time#

The first problem I encountered is how to select the aging documents! It turned out the best way to do this is to compare numbers as opposed to dates. This post helped me understand what the problem is and how to go around doing it properly. I ended it up using the built-in time stamp value stored as meta data in every DocDB document i.e. _ts. This may or may not work for every case. In my case my collection document date i.e. eventDate is actually the real UTC time ....so it was no problem. If this is not the case, you many need to store your own time stamp (in addition to the date) so u can do the query to pull the aging documents based on time.

so this query does exactly that:

Notice how I am using the Epoc time for my aging time stamp. The DateTime extension is written this way:

Partition Key#

My collection was partitioned over a value in the document i.e. source, but I wanted to trim all aging documents across all partitions...not against a single partition. So I used this query options to force the query to span multiple partitions:

Deletion#

Finally, I wanted to loop through all aging documents and delete:

Query#

Please note that the query that I used above uses a projection to get only the document link and the partition key....we really do not need the entire document:

Also please note that I am using the VALUE modifier in the query so to force DocDB to return the value only. This will return a payload that looks like this:

If I don't include the VALUE modifier, I get this:

I chose the first one :-)

Deletion#

Finally, we pull the documents and delete one at a time:

Initially, I only got the document link from the query thinking that this was the only requirement. So I did something like this:

This did not work! I needed to pass the partition key....this is why i changed the query to a projection so I can get the partition key. In my case the partition key is the source. There is a comment in this post that gave me a clue that the request option must include the partition key.

Thank you for reading! I hope this helps someone.

In this post, I just used the Service Fabric team article https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-creation-via-arm to create a PowerShell script that will do the entire deployment. I also downloaded all the required helper PowerShell modules and placed them in one repository so it would be easier for others to work with the deployment.

Here are some of my notes:

• The account you use to log in to Azure with must be a Global admin.
• In case of errors during deployment, please check the Azure Activity Logs. It is pretty good and provides a very useful insight to what went wrong.
• After a deployment is successful, you can modify the ARM template and re-deploy. This will update the cluster. For example, if you added a new LN port and re-deployed using the PowerShell script, that new port will be available.
• To connect to a secure cluster, use this guide: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-connect-to-secure-cluster
• Connect to the cluster using the browser https://your-cluster.westus.cloudapp.azure.com:19080/Explorer/index.html will cause a certificate error. This is expected as the script uses a self-signed certificate. Just proceed.
• To log in to the fabric explorer requires that you complete the steps where you go to the AD in which the cluster belongs to, select the app that was created and assign an admin role to it as described in the above article. This must be done from the classic portal.
• To connect using PowerShell, use Connect-ServiceFabricCluster -ConnectionEndpoint ${dnsName}:19000 -ServerCertThumbprint "6C84CEBF914FF489551385BA128542BA63A16222" -AzureActiveDirectory. Please note that, similar to the browser, this requires that the user be assigned as in the previous step.
• Please note that securing the cluster does not mean that your own application endpoint is secured. You must do whatever you need to do to enable HTTPs in your own application and provide some sort of token authentication.
• I noticed that the only VM size that worked reliably was the Standard_D2. Anything less than that causes health issues due to disk space, etc. I heard from Microsoft here that they are working on ways to reduce the cost of the VMs, particularly by allowing us to use smaller VMs and still get the reasonable reliability/durability levels, which would help reduce costs without sacrificing the safety or uptime of our service.

If a Web App is deployed on Azure, both the App Insights and Web Apps offer a utility that can hammer the app's endpoints from different regions. While this functionality is quite nice and comes bundled in, it is considered an alert-based system and is slightly rudimentary as one cannot customize the test or get access to the full results easily. This post describes an alternative approach that uses Azure Functions or Service Fabric to implement a web test that can test and endpoint and report its test to PowerBI in real time.

What I really wanted to do is to conduct a web test for a duration of time from different regions against a new product's endpoints at launch time and immediately view the test results with executives.

Azure Functions#

Briefly, here is what I decided to do:

• Use PowerShell to provision resource groups in four different regions. Each resource group contains an Azure Function App that loads its source code from a code repository.
• Use source code changes to trigger updates to all the functions apps at the same time.
• Configure the Azure Functions App to test a Web URL or a collection. This test can be sophisticated because we have the full power of an Azure Function to conduct the test. In other words, the test can be running through a use case scenario as opposed to just posting to a URL, for example.
• Auto-trigger the Azure Functions by a timer (i.e. configurable) to repeat the test.
• Report the source (i.e region), duration, URL, date time and status code to a PowerBI real-time data set at the end of every test iteration.
• Create a real-time visualization to see, in real-time, the test results.
• Use PowerShell script to de-provision the resource groups when the test is no longer needed.

The source code can be found here.

Macro Architecture#

Azure Resource Group Template#

I downloaded an Azure Function template and modified it to suit my needs. The main thing in the template is that it defines the repository URL and branch from which the source code is to be imported and the definitions of the app strings:

Here is the entire template:

Here is the template parameters file:

PowerShell Script#

The PowerShell script is the main driver that orchestrates the deployment and of the different Azure Functions to different resource groups. Here is the complete script:

Once you run the deployments, u will see something like this in your subscription resource groups:

PowerBI Real-Time Dataset#

Using this nifty feature in PowerBI, I defined a real-time dataset that looks like this:

This gave me a URL that I can use from Azure Functions to pump data into this Real-time dataset. Also please note that I enbaled the historic data analysis to allow me to report on the data and visualize it in real-time and beyond.

Azure Function Source Code#

Finally, the Azure Function source code that conducts the test and reports to PowerBI:

Results Visualization#

If we deploy the Azure Functions and collect the results in PowerBI, we can get real-time results that look like this:

Hopefully this visualization helps executives to see a clear indication that the product launch is not that successful :-)

Service Fabric#

I also wanted to mantion that Azure Service Fabric could also be used to conduct a web test from hammering the test site from multiple instances. Briefly, here is what I thought of doing:

• Create a Web Test Application Type. The Service Fabric app contains a single stateless service which does work on its RunAsync method.
• Use a PowerShell script to instantiate multiple tenants (or named applications): one for each region. Please note that the tenants simulate the different regions as they are all sitting in the same cluster!
• Update (i.e increase or decrease the number of the stateless service instances) each tenant (or named application) independently. This means that, unlike the Azure Functions, the system under-test can be hammered from multiple instances within the same tenant if need be.
• Write sophisticated tests within Service Fabric because the stateless service is doing it and the service can have a lot of configuration to govern that process.
• Report the source (i.e region), duration, URL, date time and status code to a PowerBI real-time data set at the end of every test iteration.
• Create a real-time visualization to see, in real-time, the test results.
• Use PowerShell script to de-provision the resource groups when the test is no longer needed.

I will publish the Service Fabric web test solution in a different blog post. For now, here is the macro architecture that I am thinking about:

I am compiling notes when working with Service Fabric from the folks at Microsoft! In this post, I will enumerate a few things that I ran into which I think might be helpful to others who are learning the environment as well. This will not be a complete list ....so I will add to it as I go along.

Actor Turn-based Concurrency#

Actors are single threaded! They only allow one thread to be acting on them at any given time. In fact, in the Service Fabric terminology, this is referred to as Turn-based treading. From my observation, it seems that this is how the platform and the actors

What happens to the clients who are calling the actors? If two clients are trying to access an actor at the same time, one blocks until the actor finishes the first client method. This is to ensure that an actor works on one thing at a time.

Let us say, we have an actor that has two methods like this:

If you call GetThing1 from one client and immediately call GetThing2 from another client (or Postman session), the second client will wait at least 15 seconds to get the string thing2 response.

Given this:

• I think it is best to front-end actors with a service that can invoke methods on actors when it receives requests from a queue. This way the service is waiting on actors to complete processing while it is in its RunAsync method.
• It is important to realize that actors should really not be queried and that actors should employ several things:
  • Backup state to an external store such as DocumentDB or others. This way the external store can be queried instead.
  • Aggregate result externally perhaps via Azure Function into some outside store so queries could run against this store
  • Aggregate result to an aggregator actor that can quickly respond to queries which will relieve the processing actors from worrying about query requests.

Actor Reminders#

Actor Reminders are really nice to have. In the sample app that I am working on, I use them to schedule processing after I return to the caller. Effectively they seem to give me the ability to run things asynchronously and return to the caller right away. Without this, the actor processing throughput may not be at best if the the processing takes a while to complete.

In addition to firing a future event, they do allow me to pack an item or object that can be retrieved when the reminder triggers. This makes a lot of scenarios possible because we are able to pack the item that we want the reminder to work on.

Please note, however, that when a reminder is running in an actor, that actor cannot respond to other method invocation! This is because the platform makes sure that there is only a single threaded operating on an actor at any time.

The best way I found out to schedule reminders to fire immediately is something like this:

When the reminder triggers, ReprocessReminder is called to process the item that was packed within the reminder: ObjectToByteArray(item). Here are possible implementation of packing and unpacking the item:

Actor Interface#

From this article: The arguments of all methods, result types of the tasks returned by each method in an actor interface, and objects stored in an actor's State Manager must be Data Contract serializable. This also applies to the arguments of the methods defined in actor event interfaces. (Actor event interface methods always return void.)

In my actor interface, I had many methods and everything was working great until I added these two methods:

If you to compile a Service Fabric solution that has an interface that looks like the above, you will be met with a very strange compilation error:

What? What is that? Why? After hours, it turned out you can actually turn off this error. From the above Stack Overflow post:

By changing the project file .csproj of the project containing the actors and setting property:

So this tool can be disabled!! But still why is this happening? It turned out that the actor interfaces may not have overridden methods!! So the tool was complaining about the interface containing just that i.e. overridden methods. If the above interface is changed to the below, everything will work well:

In addition, the actor event methods may not return anything but void. So if you have something like this, you will get the same FabActUtil.exe error:

I am hoping to add to this post as I go along. Hopefully this has been helpful.

I use Wyam to statically generate this site. This is to document the steps I take to run Wyam locally, test my posts and push the site to GitHub:

I have a directory structure that looks like this:

The Input directory looks like this:

I place my posts in Markdown in the posts directory.

The run.cmd contains the following:

This assumes that:

• I have Wyam-Exec directory two directories above my blog directory
• I have different Wyam versions I can test with. For example, I am using a beta version here

The config.wyam contains the following:

This drives the site's metadata.

While I am in my blog directory, I open up a command box and type:

This starts Wyam, launches its internal web site listening on port 5080 and it also monitors the input directory for any changes. I add my blogs, edit them and test locally using the 5080 site.

Once I am satisfied, I copy the content of the output directory:

and paste it into my site GitHub pages structure and replace all files. To publish to GitHub, I issue the following git commands:

This publishes my changes to GitHub pages.