Syslog

From Halfface
Revision as of 14:27, 20 October 2008 by Ekaanbj (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Adding a site or host to the event logging system is covered in the Site Integration Guide

Site Integration Guide

A detailed guide for joining the syslog monitoring scheme is available here

Goals

  • Capture all systems' exception logs.
  • Capture selected application error logs.
  • Transmit these messages reliably and securely.
  • Forward these messages to a central server for storage, analysis and response.
  • Track event issues and inform the relevant parties.

System Architecture

Logging services

All *nix systems come pre-installed with the BSD distributed syslog(3) logging service. This technology takes messages generated by applications and the kernel and write them to log files located in a configurable but operating system dependent location. These messages may also be forwarded, via the unreliable UDP transport, to a remote host for further processing. This is the system that was in use prior to this syslog project.

It was decided that the default syslog service was not flexible or reliable enough to base this project on so a replacement was found.

This was the syslog-ng service, which offers many advantages over the stock BSD syslog service.

  • TCP is used as the transport to ensure reliable message transfer.
  • There is a flow-control mechanism to help reliability under heavy load.
  • Syslog-ng provided extensive filtering and log rewrite facilities to categorise the incoming messages.
  • Syslog-ng understands timezones.
  • Syslog can import the log file of any program and convert these entries into syslog messages.

System roles

Each host within the Infineon organisation will act in one of three roles.

  1. Data collection node
    This is the most common form of host. It is responsible for capturing messages originating on the host itself. These messages come from two different sources; the syslog(3) logging service and from the monitoring of application log files. These events will be logged to a file, as in the default syslog implementation but will also be forwarded downstream for further processing.
    There is a Windows service available that allows an NT system to become a source of syslog messages. These messages are not yet fully integrated into the syslog project but the infrastructure is in place to capture and react to Microsoft events.
  1. Data aggregation node
    These hosts, one per geographic site, are responsible for receiving all the messages generated by that site and forwarding them downstream for further processing. These may be refered to as site hubs in other sections of this document.
    These aggregation nodes are also able to receive message from the old syslog service via the UDP transport. They are therefore able to offer backward compatibility with legacy systems that cannot be modified.
    A DNS alias, local to each site, should be created that names this host syslog.site.infineon.com.
  1. Data analysis node
    This host is the final destination for all syslog messages within Infineon as a whole or a geographical region, such as Europe. It is here that the data is analysed and an appropriate response is determined. This machine also provides the graphical front-end with which systems administrators interact.

Data collection node

Syslog messages are captured by the syslog-ng, or syslog process on legacy machines, and written to log files. A subset of these messages are then forwarded downstream to the site hub. Syslog-ng is also able to read application logs and inject their contents into the message stream. The following, possibly incomplete list of applications have been identified as producing logs that should be read and forwarded downstream to the analysis node.

  • Samba
  • Door Control system
  • License daemons
  • Clearcase system.
  • LSF
  • Apache

The location of the log files is operating system dependent under syslog; on Linux they are placed in /var/log and on Solaris they are placed in /var/adm. Syslog-ng now always placed them in /var/log, using the Linux standard.

Syslog-ng does some local filtering to remove messages that are of no interest to the central servers. For instance it does not forward any cron reports. However all messages are forwarded from the syslog service so uninteresting messages must be removed by the site hub for legacy systems.

If the system to be monitored is Windows based then the snare service should be installed. The snare server is able to pre-filter the event log messages that it incorporates into the syslog datastream and is also able to apply the required ACLs to the NT filesystem to generate the events that it wishes to capture. For instance, if file modifications to the c:\windows directory are to be monitored then the daemon will apply the correct ACLs to the files in c:\windows. It is possible that the system policies editor will interfere with this so this needs some investigation.

The local logs are directed to the following files

  • /var/log/messages
    This is the default location for all messages that are not directed to one of the other, more specialised, log files.
  • /var/log/secure
    This file holds messages referring to authorisation issues. For instance, it captures events associated with incorrect logins, PAM failures and LDAP binding faults, etc.
  • /var/log/maillog
    The mail transport service places its output in this file.
  • /var/log/cron
    Each cron job run is logged to this file together with any output that the program or cron generated.
  • /var/log/spooler
    This file will not contain many messages as its contents are produced by the uucp and news system, neither of which are used in this organisation.
  • /var/log/boot.log
    Under Linux this file will contain the messages produced during the boot sequence. On a Solaris host it will not contain anything.

Syslog-ng is controlled by the /etc/syslog-ng.conf file. An example data-collection node configuration file can be found here.

Data aggregation node

These hosts, there should be one per site, are responsible for collecting all the messages that are generated by the individual syslog and syslog-ng processes on that site. The data is then filtered to remove messages that are not of interest and then forwarded downstream for further processing. These hosts will always use the syslog-ng service for this purpose.

The host chosen for this role should have a DNS alias of syslog.site.infineon.com.

Note that this node will also contribute to the message stream as it also acts as a data collection node.

As syslog-ng captures authentication problems and can potentially be required to transport confidential information. It is therefore necessary to encrypt this traffic. This paper was the basis of this work.

Using ssh it is possible to build a compressing encrypted tunnel between any two hosts. This can be created by executing the following command on the central server; syslog.intra.infineon.com. 

ssh -nNTx -R 8728:localhost:8805 syslog.site.infineon.com > /dev/null 2>&1

This will make the port syslog.intra.infineon.com:8805 appear as port 8728 on syslog.site.infineon.com.

If this command is executed as a entry in the inittab of a Solaris/Linux host then the init process will act as a watchdog and will reconnect the tunnel should the connection drop when the machine is at runlevel 3 or 5. 

log1:35:respawn:/usr/bin/ssh -nNTx -R 8728:localhost:8805 syslog.site.infineon.com > /dev/null 2>&1

A different port was chosen at each end so that the ssh tunnels could be chained without conflict. At some point in the future even messages internal to a site may be transmitted down these encrypted tunnels.

There will be a line in /etc/inittab for each site providing a secured channel to that location.

An example /etc/syslog-ng.conf for a Data Aggregation node is provided here.

A Data Analysis and Data Agreggation node

Data analysis node

The purpose of the Data analysis node is twofold

  1. Detect and log real-time events, providing an audit log for off-line analysis.
    The events that this system attempts to capture fall into one of four categories
    • Configuration related events
    • Hardware related events
    • Resource related events
    • Security related events
    A more detailed list covering these categories can be found here.
    This functionality is provided by the [Simple Event Correlation] tool.
  2. Provide an issue tracking mechanism to capture the activities associated with an event.
    This functionality is provided by an instance of the [Resource Tracker] ticketing system.

Note that this node will also contribute to the message stream as it also acts as a data collection node

Message stores

In addition to the log files created in /var/log this node will also have a directory called /var/log/syslog-ng which is populated with a directory named for every host that contributes to the message stream. Within each of these directories are kept files with the name YYYYMMDD. E.g. There will be a file /var/log/syslog-ng/brslcs06/20071009. These files are created at midnight and filled with a day's message stream from that host. The system is configured to hold four weeks data for each host, the last three of which will be compressed via bzip2. These constitute the audit logs and are only consulted manually by humans.

The Central Syslog server also writes out a log named /var/log/syslog-ng/allmessages that contains all the messages from all hosts. This is the source file from which events are extracted.

Simple Event Correlator(SEC)

This software component, which is a perl program, is responsible for reading in events from the /var/log/syslog-ng/allmessages file and noticing interesting events or sequences of events. It is capable of drawing conclusions from message sequences originating on different hosts and at different times. This is a fairly complex program and its configuration and testing is discussed in more detail here.

SEC is located in /opt/sec-2.4.1 on a dedicated Linux node. It reads its configuration files from /etc/sec.d, which contains Config.sec.conf, Hardware.sec.conf, Resource.sec.conf and Security.sec.conf. The program runs under the logsurf account and emails its conclusions to the logmon account.

The Simple Event Correlator emails any events that it has been configured to notice to an instance of the Resource Tracker Ticketing system. These events will have been allocated to one of the four event categories; configuration, hardware, security and resource and given a priority rating.

Resource Tracker(RT3)

After the data has been collected and analysed the system requires that administrators respond to the events in a timely manner. This is accomplished by emailing the events that SEC has determined are of interest to an [RT3] incident monitoring system.

An RT3 system provides

  • Authenticated hierarchical logins
  • Web based interaction
  • Email based interaction. Useful for event insertion!
  • Event tracking throughout its lifetime.
  • Automatic event escalation
  • Automatic event assignment
  • MySQL based histories
  • Scripted customised actions
  • Structured queries through a web interface.
  • PDF Graphic reporting module

This product is generally used as a bug tracking system but it is also able to be used for a variety of functions.

The RT3 system is configured to read the email generated by the Simple Event Correlator and decode the special subject header to extract the event's type and priority. The site from which the message originated is then used to determine which queue the event should be placed in. There is one queue for each geographical location and this lets administrators focus primarily on their own site. Each administrator who will monitor the events for a site is required to add themselves as a watcher for their site's queue . This will ensure that they will be emailed with the details of any event that occurs on their site.

The RT3 system is installed in /opt/rt3 and is configured by the /opt/rt3/RT_SiteConfig.pm file. 

# Any configuration directives you include  here will override 
# RT's default configuration file, RT_Config.pm
#
# To include a directive here, just copy the equivalent statement
# from RT_Config.pm and change the value. We've included a single
# sample value below.
#
# This file is actually a perl module, so you can include valid
# perl code, as well.
#
# The converse is also true, if this file isn't valid perl, you're
# going to run into trouble. To check your SiteConfig file, use
# this comamnd:
#
#   perl -c /path/to/your/etc/RT_SiteConfig.pm 

Set( $rtname, 'infineon.com');
Set($WebBaseURL , "http://syslog.intra.infineon.com:80");
Set($WebPath , "");
### Change $DatabasePassword to something other than the default "rt_pass"
### you don't need to create this user yourself RT will create it while next step
### Set($DatabaseUser , "rt_user"); # Only uncomment if you change from the default "rt_user";
Set($DatabasePassword , "******");
Set($Organization , "infineon.com");
Set($RTAddressRegexp , '^logmon\@infineon.com$');
Set($CanonicalizeEmailAddressMatch , '@brs\.infineon\.com$');
Set($CanonicalizeEmailAddressReplace , '@infineon.com');
Set($CorrespondAddress , 'logmon');
Set($CommentAddress , 'logmon_Comment');
Set($MailCommand , 'sendmail');
Set($SendmailArguments,"-oi -t -ODeliveryMode=b -OErrorMode=m");
Set($WebExternalAuth , 1);
Set($WebFallbackToInternalAuth , 1);
Set($DefaultSummaryRows, 20);
1;

The email system must also be updated to forward all email directed at the logmon account into a mail gateway program. This excerpt from the /etc/mail/aliases file shows how this may be done. 

 logmon: "|/opt/rt3/bin/rt-mailgate --queue Bristol --action correspond --url http://syslog.intra.infineon.com"
 logmon-comment: "|/opt/rt3/bin/rt-mailgate --queue Bristol --action comment --url http://syslog.intra.infineon.com"

The site mail administrator is also required to ensure that emails directed to logmon are forwarded to the correct host.

Note that the Bristol queue has a special scrip enabled on it that forwards events for other sites to the appropriate queue.

Additional details about the customisation of the RT instance are provided here

The Solaris and Linux build

The Solaris and Linux build have been updated to include several packages to support the syslog project. There are also packages that are not installed by default that generate useful syslog output such as Sun's cediag. (This package runs a diagnostic scrub across all DIMMS and reports any anomalies.) These packages should also be added to the client and server builds.
Please update this list with any packages that you think should be added to the client builds

Solaris packaging

An attempt was made to provide the Solaris syslog-ng package as statically linked binary but this was not possible due to the following reasons. The Solaris version of the syslog-ng usss internationalization support that looks up strings in a text library. The binary therefore require support services which cannot be linked statically so the Solaris pkg will have to contain this text database. As this makes the package contain a binary and a text library lookup area it would make little sense to link the whole program statically. The final package holds several libraries but stores them in a single location so that it does not interact with /opt/TWWfsw and other library areas after the system has gone multi-user and mounted other packages. It installs into /opt/syslog-ng_2.0.5, which contains .../bin, .../share, .../lib, etc

The Solaris package file can be found here

Remember to update /etc/syslog-ng.conf with one of these, based upon the host's role.

Linux packaging

RedHat 3 requires these two packages to support syslog-ng

RedHat 4 requires these two packages to support syslog-ng

RedHat 2 packages to follow.

The configuration files must then be updated manually or via puppet from the [svn repository]. The three type are central, (for the master syslog server,) site, (for the host responsible for forwarding syslog streams from a site,) and host, (a simple drop-in replacement for the syslog service.)

You will probably need http://svn.klu.infineon.com/repos/AdminToolKit/trunk/conf/push/standard/etc/syslog-ng.conf.host.RedHat. Also remember to chkconfig --levels 2345 syslog off if installing manually.

Central Server System Specification

Using the experience gained during this investigation the specification for the target system should be

  • Quad core AMD64 hardware.
    Highest Performance is not required as the number of cores is more important. The software system has a large grain parallelism that would allow us to run the computationally intensive SEC processes in parallel. By default sec reads all four configuration classes and then processes the allmessages file to generate its event messages. It should be possible to run four instances of sec each only processing one of the event class configuration files. However, it may be that some cores should be left free to process the RT requests via Apache. Tuning will be an ongoing task.
  • 3GB+ memory.
    The SEC processes have a high memory requirement as they have to keep an in-core image of all the outstanding events that are being correlated. I have observed a peak memory size of 75MB for ~60 hosts this would lead to ~1.2GB for 1000 hosts. Allowing for other processes on the server and rule database extensions a value of 3GB+ should be adequate. However the DIMM sockets should not be fully populated to allow for errors in this calculation.
  • Twin drives.
    The system should be configured with a level-1 raid drive. There is no need for particularly fast disk IO on this system as the data rate is not very high, just continuous. The raid is just to provide high availability. Given current disk prices the system should provide 72GB of user space(2*72GB)
  • RedHat Enterprise Linux 4.
    A GoldImage 4 machine offers all the features needed to support SEC and RT3. Note that Solaris would be able to provide the required features but is much more difficult to configure.
  • Software requirements beyond a standard GI4 client
    • sec-2.4.1-1.0.el4.ifx.noarch.rpm
      • configured by puppet with files in svn
    • rt-3.6.4-1.0.el4.ifx.x86_64.rpm
      • configured by puppet with files in svn
    • Required RHEL4 packages
      • httpd-devel
      • apr-devel
      • mysql
      • mysqlclient10
      • mysql-server
      • mysql-devel
      • gcc
      • sendmail-cf
      • mod_perl
      • freetype-devel
      • gd-devel
      • libjpeg-devel
      • libpng-devel
      • xorg-x11-devel
      • perl-* (yes, all the perl modules)
    • mod_fastcgi_2.4.2-1.0.el4.ifx.x86_64.rpm
    • dag.wieers.com packages. This revision or newer.
      • perl-Apache-Session-1.80-1.2.el4.rf.noarch.rpm
      • perl-Apache-Session-1.81-1.el4.rf.noarch.rpm
      • perl-Archive-Tar-1.32-1.el4.rf.noarch.rpm
      • perl-Cache-Simple-TimedExpiry-0.27-1.el4.rf.noarch.rpm
      • perl-Calendar-Simple-1.17-1.el4.rf.noarch.rpm
      • perl-Class-Container-0.12-1.2.el4.rf.noarch.rpm
      • perl-Class-Data-Inheritable-0.06-1.el4.rf.noarch.rpm
      • perl-Class-ReturnValue-0.53-1.2.el4.rf.noarch.rpm
      • perl-Class-Singleton-1.03-1.2.el4.rf.x86_64.rpm
      • perl-Clone-0.22-1.el4.rf.x86_64.rpm
      • perl-Convert-BinHex-1.119-2.2.el4.rf.noarch.rpm
      • perl-Crypt-DES-2.05-3.2.el4.rf.x86_64.rpm
      • perl-Crypt-SSLeay-0.56-1.el4.rf.x86_64.rpm
      • perl-DateTime-0.2901-1.2.el4.rf.x86_64.rpm
      • perl-DateTime-Format-Mail-0.30-1.el4.rf.noarch.rpm
      • perl-DateTime-Format-W3CDTF-0.04-1.el4.rf.noarch.rpm
      • perl-DateTime-Locale-0.22-1.2.el4.rf.x86_64.rpm
      • perl-DateTime-TimeZone-0.46-1.el4.rf.x86_64.rpm
      • perl-DBI-1.58-2.el4.rf.x86_64.rpm
      • perl-DBIx-DBSchema-0.31-1.el4.rf.noarch.rpm
      • perl-DBIx-SearchBuilder-1.49-1.el4.rf.noarch.rpm
      • perl-Devel-StackTrace-1.15-1.el4.rf.noarch.rpm
      • perl-Exception-Class-1.23-1.el4.rf.noarch.rpm
      • perl-FCGI-0.67-1.2.el4.rf.x86_64.rpm
      • perl-GD-2.35-1.el4.rf.x86_64.rpm
      • perl-GD-Graph-1.43-0.2.el4.rf.x86_64.rpm
      • perl-GD-Graph-1.43-1.2.el4.rf.noarch.rpm
      • perl-GD-Text-Util-0.86-0.2.el4.rf.x86_64.rpm
      • perl-GD-Text-Util-0.86-1.2.el4.rf.noarch.rpm
      • perl-Hook-LexWrap-0.20-1.el4.rf.noarch.rpm
      • perl-HTML-Format-2.04-1.2.el4.rf.noarch.rpm
      • perl-HTML-Mason-1.3200-1.el4.rf.noarch.rpm
      • perl-HTML-Parser-3.55-1.el4.rf.x86_64.rpm
      • perl-HTML-Scrubber-0.08-1.2.el4.rf.noarch.rpm
      • perl-HTML-Tagset-3.10-1.el4.rf.noarch.rpm
      • perl-HTML-Tree-3.23-1.el4.rf.noarch.rpm
      • perl-HTTP-Server-Simple-0.27-1.el4.rf.noarch.rpm
      • perl-HTTP-Server-Simple-Mason-0.09-1.el4.rf.noarch.rpm
      • perl-IO-Socket-INET6-2.51-1.2.el4.rf.noarch.rpm
      • perl-IO-Socket-SSL-1.07-2.el4.rf.noarch.rpm
      • perl-IO-stringy-2.110-1.2.el4.rf.noarch.rpm
      • perl-IO-Zlib-1.05-1.el4.rf.noarch.rpm
      • perl-LDAP-0.34-1.el4.rf.noarch.rpm
      • perl-Locale-Maketext-Fuzzy-0.02-1.2.el4.rf.noarch.rpm
      • perl-Locale-Maketext-Lexicon-0.64-1.el4.rf.noarch.rpm
      • perl-Log-Dispatch-2.18-1.el4.rf.noarch.rpm
      • perl-Mail-Sender-0.8.13-1.el4.rf.noarch.rpm
      • perl-Mail-Sendmail-0.79-1.2.el4.rf.noarch.rpm
      • perl-MailTools-1.77-1.el4.rf.noarch.rpm
      • perl-MIME-Lite-3.01-2.2.el4.rf.noarch.rpm
      • perl-MIME-tools-5.420-2.el4.rf.noarch.rpm
      • perl-Module-Versions-Report-1.03-1.el4.rf.noarch.rpm
      • perl-Net-Daemon-0.43-1.el4.rf.noarch.rpm
      • perl-Net-DNS-0.61-1.el4.rf.x86_64.rpm
      • perl-Net-IP-1.25-1.el4.rf.noarch.rpm
      • perl-Net-SNMP-5.0.1-1.2.el4.rf.noarch.rpm
      • perl-Net-SNMP-5.2.0-1.2.el4.rf.noarch.rpm
      • perl-Net-SSLeay-1.30-1.el4.rf.x86_64.rpm
      • perl-Params-Validate-0.86-1.el4.rf.x86_64.rpm
      • perl-PlRPC-0.2020-1.el4.rf.noarch.rpm
      • perl-Regexp-Common-2.120-1.2.el4.rf.noarch.rpm
      • perl-Socket6-0.19-1.2.el4.rf.x86_64.rpm
      • perl-Term-ReadKey-2.30-2.el4.rf.x86_64.rpm
      • perl-Text-Autoformat-1.13-1.2.el4.rf.noarch.rpm
      • perl-Text-Quoted-2.02-1.el4.rf.noarch.rpm
      • perl-Text-Reform-1.11-1.2.el4.rf.noarch.rpm
      • perl-Text-Template-1.44-1.2.el4.rf.noarch.rpm
      • perl-Text-WikiFormat-0.78-1.el4.rf.noarch.rpm
      • perl-Text-Wrapper-1.000-1.2.el4.rf.noarch.rpm
      • perl-TimeDate-1.16-1.2.el4.rf.noarch.rpm
      • perl-Time-modules-2003.1126-1.2.el4.rf.noarch.rpm
      • perl-Tree-Simple-1.17-1.el4.rf.noarch.rpm
      • perl-UNIVERSAL-require-0.11-1.el4.rf.noarch.rpm
      • perl-Want-0.14-1.el4.rf.x86_64.rpm
      • perl-XML-Dumper-0.81-1.el4.rf.noarch.rpm
      • perl-XML-NamespaceSupport-1.09-1.2.el4.rf.noarch.rpm
      • perl-XML-RSS-1.22-1.el4.rf.noarch.rpm
      • perl-XML-SAX-0.16-1.el4.rf.noarch.rpm
      • perl-XML-Twig-3.29-1.el4.rf.noarch.rpm
      • python-urlgrabber-2.9.7-1.2.el4.rf.noarch.rpm
      • spamassassin-3.2.3-1.el4.rf.x86_64.rpm

These packages have been added to a yumgroup called Central syslog server + RT3.

Central Server Deployment

Hardware has been allocated at the Klagenfurt site for the deployment of the European Central Server, syslog.intra.infineon.com. It will be hosted by a virtual machine running under VMware and will be initially provisioned with 4 cores and 10GB of SAN disk.

Post Install Operations

The system can be built using puppet however some additional configuration was required to customize the RT instance for Infineon Event tracking.

  • Removed or edited all global scrips that replied to requestor
    These can be found at Configuration->Global->Scrips
    The requestor will always be user logsurf. No Emails should be sent to this account.
  • Changed the "real name" for user root from "Enoch Root" to "Root"
    This can be found at Configuration->Users->root
  • Created a Global Ticket Custom Field called "Event Type"
    This can be found at Configuration->Global->Custom Fields->Ticket
    When an event is generated by SEC it is placed into one of four categories.
    • Configuration issues
    • Hardware issues
    • Resource issues
    • Security issues
    The custom field is a Combo box type with a name and description matching the event types
  • Created a Global Ticket Custom Field called "Host"
    This can be found at Configuration->Global-<Custom Fields->Ticket
    This field holds the name of the host on which the event originated.
    The custom field type is "Enter one value"
  • The General queue was renamed Bristol
    This can be found at Configuration->Queues->General
  • A queue was created for all other participating sites.
    This can be found at Configuration->Queues->New Queue
  • Two On-Create scrips were applied to the Bristol queue
    This can be found at Configuration->Queues->Bristol->Scrips->New Scrip
    Scrip one
    • Description: Set Custom Field From Subject
    • Condition: On Create
    • Action: Set Custom Fields By Subject
    • Template: Global Template: Blank
    • Stage: TransactionCreate
    Scrip two
    • Description: Set Queue From Subject Field
    • Condition: On Create
    • Action: Set Queue By Subject
    • Template: Global Template: Blank
    • Stage: TransactionCreate
  • A user account was created for all interested parties
    This can be found at Configuration->Users->New User
  • A group account was created for each site or queue called <site>-it
    This can be found at Configuration->Groups->New Group
  • A user that wishes to be alerted to events on the Bristol queue would place themselves in the Bristol-it group
    This can be found at Configuration->Groups->Bristol-it->Members
  • A group called it-access was created into which all the users were placed.
    This can be found at Configuration->Groups->New Group and Configuration->Groups->it-access->Members
  • This group was granted system rights to access and modify tickets.
    This can be found at Configuration->Global->Group Rights

Future Improvements

Initial project Investigation

The inital project investigation plan can be found here

Notes

Message Matrix

The matrix is very difficult to maintain in Wiki format so I have placed it in this sharepoint

Severity levels

LOG_EMERG
       A panic condition.

LOG_ALERT
       A condition that should be corrected immediately, such as a corrupted system database.

LOG_CRIT
       Critical conditions, such as hard device errors.

LOG_ERR
       Errors.

LOG_WARNING
       Warning messages.

LOG_NOTICE
       Conditions that are not error conditions, but that may require special handling.

LOG_INFO
       Informational messages.

LOG_DEBUG
       Messages that contain information normally of use only when debugging a program.

Meetings

syslog 8/8/2007

syslog 16/8/2007

sylog 19/9/2007

Retrieved from "https://halfface.se/wiki/index.php?title=Syslog&oldid=2788"