Technical Insights: Azure, .NET, Dynamics 365 & EV Charging Architecture

Month: October 2015

Logging in .NET – Elastic Search, Kibana and Serilog

I’ve been using log4net in the past and I found it quite useful as it is ready to use out of the box. In my last workplace, we are using SPLUNK and its amazing as I’m able to troubleshoot production issue by looking at the trend and activities. You can do query based and filtering the log and build a pretty dashboard. Downside of it is the cost for Splunk is expensive (i don’t think its for the mainstream user or small business)

So I’ve found another logging mechanism/storage/tool which is amazing!! It is called Elastic Search and its open source (well there are different subscriptions level for better supports). Pretty much Elastic Search is the engine for search and analytics

How about the GUI/Dashboard?Yes you can use Kibana. It is an open source data visualization platform that allows you to interact with data

Ok, so lets say if I have a .NET, how do I write my log to Elastic Search?You can use SeriLog. It will allow you to log structured event data to your log and you can use Serilog Elastic Search sink to integrate with Elastic Search

Serilog has different sink providers that allow you to store your log externally (beside file) including SPLUNK

I will talk more about Serilog separately in different post, stay tune!

Git – Prune your local branches to keep it in sync with remote branches

On your local branches normally you have a stale branch where it doesn’t have corresponding remote branch and you feel like you want to make it in sync with the remote branches

1. Lets start listing the remote branch first just to know what are the branches available remotely

[code language=”bash”]
$ git remote show origin
[/code]

2. Lets see our local stale branches “–dry-run” command will just display the stale branch but without deleting it

[code language=”bash”]
$ git remote prune origin –dry-run
[/code]

3. Alternatively if you want to really delete the stale branches you can run it without “–dry-run” command

[code language=”bash”]
$ git remote prune origin
[/code]

*just make sure you already committed your feature branch to the remote before doing this

Dynamic Deserialization using JsonConverter

This post is the continuation from the previous post. The previous post was in regard to casting the object dynamically. This post will explain how to deserialize dynamically from Json object

I have a json that I want to deserialize dynamically based on a specific property that defines what object it is. We can do it easily and elegantly by using JsonConverter

1. Create a custom JsonConverter
[code language=”csharp”]
public class MessageConverter : JsonConverter
{
static readonly JsonSerializerSettings SpecifiedSubclassConversion = new JsonSerializerSettings() { ContractResolver = new CamelCasePropertyNamesContractResolver() };

public override bool CanConvert(Type objectType)
{
return (objectType == typeof(WhafflMetadata));
}

public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
var jObject = JObject.Load(reader);

switch (jObject["messageType"].Value<string>())
{
case "STAFF_CREATED":
return JsonConvert.DeserializeObject<Message<StaffDetail>>(jObject.ToString(), SpecifiedSubclassConversion);

case "CITY_CREATED":
return JsonConvert.DeserializeObject<Message<City>>(jObject.ToString(), SpecifiedSubclassConversion);

default:
throw new Exception(string.Format("messageType {0} cannot be handled", jObject["messageType"].Value<string>()));
}
}

public override bool CanWrite
{
get { return false; }
}

public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotImplementedException();
}
}
[/code]

2. Cast it using your converter

[code language=”csharp”]
JsonConvert.DeserializeObject<Message>(
jsonValue,
new JsonSerializerSettings { Converters = new JsonConverter[] { new MessageConverter() } });
[/code]

Automapper – Dynamic and Generic Mapping

In Automapper, we normally have 1 to 1 mapping defined but I have a case whereas the incoming stream as a json payload which then I cast it as a dynamic (using JObject parse) and in one of the property within the payload it defined which object that it needs to cast into. Lets take a look at the sample below

Input
Json payload to create a city

[code language=”javascript”]
{
"requestId": "C4910016-C30D-415C-89D3-D08D724429A6",
"messageType": "CITY_CREATED",
"categoryName": "categoryA",
"metadata": {
"city": "sydney",
"state": "NSW",
"postcode": "2000",
"country": "australia"
}
}
[/code]

at the same time we can also have a Json payload to create a staff

[code language=”javascript”]
{
"requestId":"C4910016-C30D-415C-89D3-D08D724429A6",
"messageType": "STAFF_CREATED",
"categoryName": "categoryB",
"staffDetail": {
"name": "fransiscus",
"dateOfBirth": "01/01/1950"
},
"location" : {
"cityId" : "1"
}
}
[/code]

So what we are doing in here, all the message will go into payload property (it can contain any object) and we add some extra information/header/metadata on the parent level
Desired Outputs

[code language=”javascript”]
{
"messageType": "CITY_CREATED",
"payload": {
"city": "sydney",
"state": "NSW",
"postcode": "2000",
"country": "australia"
},
"provider": "abc",
"providerRequestId": "C4910016-C30D-415C-89D3-D08D724429A6",
"receivedAt": "2015-09-30T23:53:58.6118521Z",
"lastUpdated": "2015-09-30T23:53:58.6128283Z",
"lastUpdater": "Transformer",
"attempt": 0
}
[/code]

[code language=”javascript”]
{
"messageType": "STAFF_CREATED",
"payload": {
"staffName": "fransiscus",
"dateOfBirth": "01/01/1950",
"cityId": "1"
},
"provider": "abc",
"providerRequestId": "C4910016-C30D-415C-89D3-D08D724429A6",
"receivedAt": "2015-09-30T23:53:58.6118521Z",
"lastUpdated": "2015-09-30T23:53:58.6128283Z",
"lastUpdater": "Transformer",
"attempt": 0
}
[/code]

To map this to a concrete class 1:1 mapping is straight forward and easy. The problem here is that the “messageType” is the one that decided which object that it should be

Automapper Configuration:

1. POCO object

abstract class that stores all the metadata

[code language=”csharp”]
public abstract class Metadata
{
public string MessageType { get; set; }

public string Provider { get; set; }

public string ProviderRequestId { get; set; }

public DateTime ReceivedAt { get; set; }

public DateTime LastUpdated { get; set; }

public string LastUpdater { get; set; }

public int Attempt { get; set; }

public List<string> Errors { get; set; }
}
[/code]

[code language=”csharp”]
public class City
{
public string CityName { get; set; }
public string State { get; set; }
public string PostCode { get; set; }
public string Country { get; set; }
}
[/code]

[code language=”csharp”]
public class StaffDetail
{
public string Name { get; set; }
public string DateOfBirth { get; set; }
public int CityId { get; set; }
}
[/code]

[code language=”csharp”]
public class Message<T> : Metadata where T : class
{
public T Payload { get; set; }
}
[/code]

2. Lets create a TypeConverter for the base class which is Metadata and from this converter it will return the derived class

[code language=”csharp”]
public class MetadataTypeConverter : TypeConverter<dynamic, Metadata>
{
protected override Metadata ConvertCore(dynamic source)
{
Metadata metadata;

var type = (string)source.messageType.Value;

switch (type)
{
case "STAFF_CREATED":
metadata = new Message<StaffDetail> { Payload = Mapper.Map<dynamic, StaffDetail>(source) };
break;
case "CITY_CREATED":
metadata = new Message<City> { Payload = Mapper.Map<dynamic, City>(source) };
break;

default: throw new Exception(string.Format("no mapping defined for {0}", source.messageType.Value));
}

metadata.ProviderRequestId = source.requestId;
metadata.Topic = string.Format("{0}.{1}.pregame",
producerTopicName,
source.categoryName ?? source.competition.categoryName);
metadata.Provider = "My Provider";
metadata.MessageType = source.messageType;
metadata.ReceivedAt = DateTime.UtcNow;
metadata.LastUpdated = DateTime.UtcNow;
metadata.LastUpdater = "Transformer";
metadata.Attempt = 0;

return metadata;
}
}
[/code]

3. Lets create a TypeConverter for the derived class which are Staff and City

[code language=”csharp”]
public class CityTypeConverter : TypeConverter<dynamic, City>
{
protected override City ConvertCore(dynamic source)
{
City city = new City();
city.CityName = source.metadata.city;
city.State = source.metadata.state;
city.Postcode = source.metadata.postcode;
city.Country = source.metadata.country;

return city;
}
}
[/code]

[code language=”csharp”]
public class StaffDetailTypeConverter : TypeConverter<dynamic, StaffDetail>
{
protected override StaffDetail ConvertCore(dynamic source)
{
StaffDetail staffdetail = new StaffDetail();
staffdetail.Name = source.staffDetail.name;
staffdetail.DateOfBirth = source.staffDetail.dateOfBirth;
staffdetail.CityId = source.location.cityId;

return staffdetail;
}
}
[/code]

3. Define the Automapper mapping in the configuration

[code language=”csharp”]
public class WhafflMessageMapping : Profile
{
public override string ProfileName
{
get
{
return this.GetType().Name;
}
}

protected override void Configure()
{
this.CreateMap()
.ConvertUsing(new MetadataTypeConverter());

this.CreateMap()
.ConvertUsing(new StaffDetailTypeConverter());

this.CreateMap()
.ConvertUsing(new CityTypeConverter());
}

private Metadata BuildWhafflMessage(dynamic context)
{
var type = ((string)context.messageType.Value);

switch (type)
{
case "STAFF_CREATED":
return new Message { Payload = Mapper.Map(context) };
case "CITY_CREATED:
return new Message { Payload = Mapper.Map(context) };

default: throw new Exception(string.Format("no mapping defined for {0}", context.messageType.Value));
}

}
}
[/code]

Powered by WordPress & Theme by Anders Norén