When you followed the previous posts (installation, log4net) you have a pretty good coverage of Logstash on your servers.

To avoid that Logstash is running on command line or even needs to be started manually upon reboot, you can run it as a Windows Service using nssm.

I decided to implement a small batch file which starts Logstash:

@echo off
cd "\Logstash"
set home="c:\logstash\sincedb"
set LS_MAX_MEM=256m
set JAVA_HOME=C:\Program Files (x86)\Java\jre7
Logstash-1.4.0\bin\logstash agent -f my-server.conf

You need to adjust the paths according to your environment.

The installation of the Windows Service with nssm is done using:

nssm install Logstash C:\logstash\startlogstash.bat

As mentioned in the log4net posting, I run decentral instances of Logstash which then forward the log entries to a central Redis instance.

This instance is also running as a Windows Service, installed in similar fashion:

nssm install "Redis Server" C:\logstash\redis\redis-server.exe

After setting up Logstash on Windows in basic manner and adding Eventlog handling (see here), you might want to make use of Logstash for your .net applications.

There are multiple solutions on the internet, including using custom appenders. I decided to take a different route, following e.g. this post.

I am using UdpAppender to efficiently push log entries to Logstash. As udp is not realiable, I furthermore decided to run one Logstash instance on each application server which then subsequently forwards the log entry to a central Logstash server. In fact, this works much like MTA daemons on Unix systems. The overhead on each machine is small and the big advantage is that I can use vanilla log4net and use Logstash without modifying my .net applications. Small disadvantage is that I have to care about Logstash on all of my servers, i.e. making sure that it is running and up-to-date.

The log4net configuration looks like:

<appender name="UdpAppender" type="log4net.Appender.UdpAppender">
 <RemoteAddress value="127.0.0.1" />
 <RemotePort value="5960" />
 <layout type="log4net.Layout.PatternLayout">
 <conversionPattern value="%date [%thread] %-5level - %property{log4net:HostName} - MyApplication - %message%newline" />
 </layout>
 </appender>

I use a dedicated input which is used to mark all entries to come from log4net:

udp { 
  port => 5960 
  codec => plain { 
    charset => "UTF-8" 
  } 
  type => "log4net" 
}

I use a custom conversion pattern which is then dissected by a grok filter:

filter {
  if [type] == "log4net" {
    grok {
      remove_field => message
      match => { message => "(?m)%{TIMESTAMP_ISO8601:sourceTimestamp} \[%{NUMBER:threadid}\] %{LOGLEVEL:loglevel} +- %{IPORHOST:tempHost} - %{GREEDYDATA:tempMessage}" }
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "message" , "%{tempMessage}" ]
        replace => [ "host" , "%{tempHost}" ]
      }
    }
    mutate {
      remove_field => [ "tempMessage" ]
      remove_field => [ "tempHost" ]
    }
}

This rule parses the log entries and fills the entries sourceTimestamp, threadid, loglevel, host, message.

We no have the messages stored on my application servers. For moving the messages to the central Logstash server, I decided to use Redis. It is an efficient database/ queueing system. Unfortunately, only an old version can be downloaded from the Redis site.
As data is only stored temporarily in Redis and bleeding-edge features are not required, this version is sufficient for my needs.

The configuration for exporting the data from the application servers is:

output {
  	redis {
	  key => "logstash-centralized"
	  codec => json {
            charset => "UTF-8"
          }
	  data_type => "list"
	  host => "my-central-server"
	}
}

I then have full options for further processing. I can either read directly from Redis or read the log entries from Redis with my central Logstash instance.

Previous posts on Logstash:

• Installation Logstash

Next posts:

• Running Logstash as a Windows Service

Ever since I discovered Logstash, I really admire what the guys at elasticsearch invented – how easy it is to use and how actually simple the application is in its core.

This post will give some hints on how to run Logstash under Windows, supporting log4net.
The post collects a number of sources and will hopefully allow you to set up Logstash successfully in your environment, too.

Event logs

Logstash 1.40 is unfortunately not coming with the eventlog gem installed per default any more.
I discovered this post describing a similar issue:

https://logstash.jira.com/browse/LOGSTASH-1918

so all you have to do is:

1. Get JRuby and add its bin directory to your path
2. Set GEM_HOME to the vendor/bundle/jruby/1.9 directory within your logstash installation
3. Clear GEM_PATH (“”)
4. gem install win32ole
5. Check that vendor\bundle\jruby\1.9\gems directory within your logstash installation contains the jruby-win32ole directory
6. Finished

Now you can use eventlog as your input:

input {
  eventlog {
    type => 'Win32-EventLog'
    logfile => 'System'
  }
}

Next posts on Logstash:

• Using Logstash with log4net
• Running Logstash as a Windows Service

Danke des Sponsorings von CKSolutions konnte ich die ZUGFeRD .net-Bibliothek um die Möglichkeit erweitern, ZUGFeRD XML-Dateien auch einlesen zu können.

Diese Funktion ist nun (nahezu) vollständig implementiert.

Daneben ist die Bibliothek nun in der Lage, aus Streams zu lesen und in Streams zu schreiben.

Die aktuelle Version der Bibliothek gibt es hier:
auf nuget: https://www.nuget.org/packages/ZUGFeRD-csharp/
auf github: https://github.com/stephanstapel/ZUGFeRD-csharp

Passend zu diesem Post gibt es bereits einige Posts in Englisch (hier, hier).

Zur einfacheren Lesbarkeit habe ich mich entschlossen, diese Information zusätzlich auf Deutsch anzubieten.

ZUGFeRD ist eine Initiative, die unter dem Dach des BMWi (Bundesministerium für Wirtschaft und Technologie) gegründet wurde. Es beruht darauf, Rechnungen im PDF/A-3-Format zu erstellen und in dieses Dokument eine xml-Datei einzubetten, die alle Rechnungsinformationen in strukturierter Form enthält. Diese xml-Datei ermöglicht zum einen das einfache Indizieren der PDF-Datei in ein Dokumentenarchiv und zum anderen auch das automatische Buchen der Rechnungen in ERP-Lösungen.

Für die Erstellung von PDF/A-3-Dateien gibt es diverse Lösungen am Markt. Besonders die Konverter von PDF nach PDF/A-3 sind sehr einfach in der Handhabung.

Die Erstellung der xml-Datei jedoch ist schwierig. Es gibt einige Anbieter, die passende Bibliotheken anbieten, jedoch sind diese in der Regel kommerziell und machen teilweise einen nicht sehr reifen Eindruck.

Daher habe ich mich entschlossen, eine eigene Bibliothek zu entwickeln, die die Erstellung von ZUGFeRD-xml-Dateien mit .net (C#, Visual Basic.net etc.) ermöglicht. Diese Bibliothek wird unter github gehostet und ermöglicht so die Mitarbeit anderer Interessierter.

Um die Nutzung in Visual Studio zu erleichtern, ist die ZUGFeRD-Bibliothek ebenfalls unter nuget verfügbar.

Das xml-Format ist relativ komplex und inhaltlich vergleichbar mit einer EDI-Rechnung. Bei der Entwicklung habe ich mich daher bemüht, den Funktionen möglichst sprechende Namen zu geben und möglichst viel Dokumentation einzutragen.

Dieses Beispiel:

InvoiceDescriptor desc = InvoiceDescriptor.CreateInvoice("471102", new DateTime(2013, 6, 5), CurrencyCodes.EUR, "GE2020211-471102");
desc.Profile = Profile.Comfort;
desc.ReferenceOrderNo = "AB-312";
desc.AddNote("Rechnung gemäß Bestellung Nr. 2013-471331 vom 01.03.2013.");
desc.AddNote("Es bestehen Rabatt- und Bonusvereinbarungen.", SubjectCode.AAA);
desc.SetBuyer("Kunden Mitte AG", "69876", "Frankfurt", "Kundenstraße", "15", "DE", "88", "4000001987658");
desc.AddBuyerTaxRegistration("DE234567890", TaxRegistrationSchemeID.VA);
desc.SetBuyerContact("Hans Muster");
desc.SetSeller("Lieferant GmbH", "80333", "München", "Lieferantenstraße", "20", "DE", "88", "4000001123452");
desc.AddSellerTaxRegistration("201/113/40209", TaxRegistrationSchemeID.FC);
desc.AddSellerTaxRegistration("DE123456789", TaxRegistrationSchemeID.VA);
desc.SetBuyerOrderReferenceDocument("2013-471331", new DateTime(2013, 03, 01));
desc.SetDeliveryNoteReferenceDocument("2013-51111", new DateTime(2013, 6, 3));
desc.ActualDeliveryDate = new DateTime(2013, 6, 3);
desc.SetTotals(202.76m, 5.80m, 14.73m, 193.83m, 21.31m, 215.14m, 50.0m, 165.14m);
desc.AddApplicableTradeTax(9.06m, 129.37m, 7m, TaxType.VAT, TaxCategoryCode.S);
desc.AddApplicableTradeTax(12.25m, 64.46m, 19m, TaxType.VAT, TaxCategoryCode.S);
desc.AddLogisticsServiceCharge(5.80m, "Versandkosten", TaxType.VAT, TaxCategoryCode.S, 7m);
desc.AddTradeAllowanceCharge(true, 10m, CurrencyCodes.EUR, 1m, "Sondernachlass", TaxType.VAT, TaxCategoryCode.S, 19);
desc.AddTradeAllowanceCharge(true, 137.7m, CurrencyCodes.EUR, 13.73m, "Sondernachlass", TaxType.VAT, TaxCategoryCode.S, 7);
desc.SetTradePaymentTerms("Zahlbar innerhalb 30 Tagen netto bis 04.07.2013, 3% Skonto innerhalb 10 Tagen bis 15.06.2013", new DateTime(2013, 07, 04));

desc.Save("output.xml");

erzeugt die xml-Datei wie sie im ZUGFeRD-Informationspaket enthalten ist. Eigentlich relativ gut leserlich. Oder?!

Today I have updated my ZUGFeRD .net library.

String parameters have been replaced wherever possible by enums to make usage more convenient and safer. This applies to e.g. tax type (VAT etc.) and tax registration scheme (FC = fiscal number, VA = VAT registration number).

Remembering all the specified EDIFACT values is a nightmare so this should enhance usability of the library a lot.

You can fork the code on github:
https://github.com/stephanstapel/ZUGFeRD-csharp

and start using the library by adding the nuget package for your .net projects (C#, Visual Basic .net) directly in Visual Studio:
https://www.nuget.org/packages/ZUGFeRD-csharp/

I recently came across a nice initiative from German government and other (semi-)state organizations for developing a standard for electronic invoices. It uses PDF/A-3 as a basis. In the PDF file, an XML file is embedded which serves as a descriptor of the PDF file. It contains information about buyer, seller, invoice header information as well as line items.

The descriptor uses existing mechanisms from e.g. UN/CEFACT. It is still developing, e.g. there are plans to extend it to Europe.

You can find more information here:

http://www.ferd-net.de/

The ZUGFeRD initiative delivers a set of sample files along with schema files and schema documentation. Unfortunately, there is no source code for creating ZUGFeRD compliant invoices.

So I decided to create such a library as a C# .net assembly:

https://github.com/stephanstapel/ZUGFeRD-csharp

This component is writting in C# but allows you to create ZUGFeRD compliant XML files using any .net language.

Update (2013-11-17):
ZUGFeRD library is now also available on nuget for plugging it in easily into your .net projects:

https://www.nuget.org/packages/ZUGFeRD-csharp/1.0.0

It is not yet complete as it does not support the following yet (in order of my personal priority):

• some enums are still passed as strings
• parse (load) existing ZUGFeRD file
• support the three ZUGFeRD profiles
• write line items

Please feel free to fork the code and issue pull requests.

The actual usage is not dead simple as a generic invoice structure itself is somewhat complicated. The library comes with a sample application which creates the same invoice file as shipping with the ZUGFeRD information package found on the initiative’s website:

InvoiceDescriptor desc = InvoiceDescriptor.CreateInvoice("471102", new DateTime(2013, 6, 5), CurrencyCodes.EUR, "GE2020211-471102");
desc.Profile = Profile.Comfort;
desc.ReferenceOrderNo = "AB-312";
desc.AddNote("Rechnung gemäß Bestellung Nr. 2013-471331 vom 01.03.2013.");
desc.AddNote("Es bestehen Rabatt- und Bonusvereinbarungen.", "AAK");
desc.SetBuyer("Kunden Mitte AG", "69876", "Frankfurt", "Kundenstraße", "15", "DE", "88", "4000001987658");
desc.AddBuyerTaxRegistration("DE234567890", "VA");
desc.SetBuyerContact("Hans Muster");
desc.SetSeller("Lieferant GmbH", "80333", "München", "Lieferantenstraße", "20", "DE", "88", "4000001123452");
desc.AddSellerTaxRegistration("201/113/40209", "FC");
desc.AddSellerTaxRegistration("DE123456789", "VA");
desc.SetBuyerOrderReferenceDocument("2013-471331", new DateTime(2013, 03, 01));
desc.SetDeliveryNoteReferenceDocument("2013-51111", new DateTime(2013, 6, 3));
desc.ActualDeliveryDate = new DateTime(2013, 6, 3);
desc.SetTotals(202.76m, 5.80m, 14.73m, 193.83m, 21.31m, 215.14m, 50.0m, 165.14m);
desc.AddApplicableTradeTax(9.06m, 129.37m, 7m, "VAT", "S");
desc.AddApplicableTradeTax(12.25m, 64.46m, 19m, "VAT", "S");
desc.SetLogisticsServiceCharge(5.80m, "Versandkosten", "VAT", "S", 7m);
desc.setTradePaymentTerms("Zahlbar innerhalb 30 Tagen netto bis 04.07.2013, 3% Skonto innerhalb 10 Tagen bis 15.06.2013", new DateTime(2013, 07, 04));

desc.Save("output.xml");

Please also see for an update of the library here.

I really like the National Geographic photo of the day page. As I only have little time to care about my Windows profile’s styling, I wanted to set these photos as my desktop background.

There are dozens of applications on the market, allowing to download and manage these images along with images from multiple other sites. These applications compete with each other in offering more and more features. While this is of course nice in general, it makes the applications less robust and less silent during booting to automatically set the current photo of the day.

During waiting for a flight at the airport, I was just fed up with one of the applications I tended to use until that point. Wifi wasn’t yet available and the application just stalled before I could finally manually (!) update the photo of the day. So I sat down for about an hour and implemented a little application for exactly this purpose:

https://github.com/stephanstapel/WallpaperDownloader

This application is designed to be really minimalistic, currently offering no user interface, only downloading the current photo of the day and then exiting. The application is robust enough the tolerate unavailable LAN and Wifi connections with automatic periodic retries.

It will then download the current photo, store it into a ‘My pictures’ subfolder and setting it as the current wallpaper.

Have fun with the application, looking forward for your enhancements!

Stepahn

In my previous blog post I wrote about how to make Graphite monitoring system run on Windows.
Coming with Graphite (respectively the Carbon component), the Graphite WebApp is availalable to view monitoring information.

This application is completely cross-platform, so it doesn’t need any adoptions to the Windows operating system.

It only requires certain third party components to be available on the system:

• Install Python 2.7 (as mentioned in the other blog post)
• Install PyPi (as mentioned in the other blog post)
• Install PythonGTK+ (for Cairo graphics) from here
• Install Django using a package from here. In my case, I used Django 1.4.2. Drop me a message if Graphite WebApp also works with newer versions of Django.
• Install twisted using:

  pip install Twisted

• Install zope.interface using:

  pip install zope.interface

The installation itself works just like described in the base Graphite installation post:

• python check-dependencies.py

• python setup.py install

You can test Graphite WebApp using Django development tools:

django-admin.py runserver --pythonpath d:\graphite\webapp --settings graphite.settings 0.0.0.0:8080

with the following important parameters:

• d:\graphite\webapp as the path where the Graphite WebApp is stored.
• The –settings parameter is required to let Django set up its database. There are two good postings for this:
  • https://answers.launchpad.net/graphite/+question/196387
  • http://stackoverflow.com/questions/9850581/django-error-when-installing-graphite-settings-databases-is-improperly-configu

Ron Cordell from RelayHealth reported that they run Graphite WebApp in IIS with  Helicon Zoo which is available for free here. I didn’t try this and if you should have experience with it, let me know and I’ll add detailed steps for it in this post.

As mentioned in my previous post, I discovered the Graphite monitoring system and ported it to Windows. That wasn’t as difficult as I initially thought so I’ll explain the steps in detail in this and more upcoming posts so that everyone interested can follow the guide easily.

I got good hints and help from Ron Cordell from Relay Health who previously ported Graphite to Windows in a similar fashion. Thanks a lot, Ron!

Prerequisites for running Graphite are:

• install Python (in my case: latest release of version 2.7)
• install pypi/ setuptools (download here)

I forked the official carbon from github and inserted the changes I made. You find the fork here:

https://github.com/stephanstapel/carbon

There are some steps that you need to perform in order to successfully run Graphite on Windows:

1. Download Carbon (here)
2. Download Whisper (here)
3. Download Graphite-web (here)
4. Install Whisper using setup.py install
   It goes into the Python libs/site-packages directory
5. Change the path in Carbon’s setup.cfg to a Windows path. In my case:
   prefix = d:\graphite
6. Run setup.py install
7. Change local data directory setting: open conf/carbon.conf and change LOCAL_DATA_DIR. Alternatively, you can set following environment variables as specified in carbon.conf file:
   • GRAPHITE_ROOT
   • GRAPHITE_CONF_DIR
   • GRAPHITE_STORAGE_DIR

After committing my changes into the branch, I issued a pull request and hope that it will be approved.

If you find anything not to be working, please let me know.

The second article will cover the Graphite-webapp. This app runs on Windows without any code change but requires some additional packages to be installed.