Syslog: Difference between revisions

From Halfface
Jump to navigation Jump to search
No edit summary
 
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
<span style="color:red"><b>Adding a site or host to the event logging system is covered in the  [[Site Integration Guide]]</b></span>
==syslog==


== <span style="color:red">Site Integration Guide</span> ==
Messages refer to a facility (<code>auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0, ... , local7 </code>) and are assigned a severity (<code>Emergency, Alert, Critical, Error, Warning, Notice, Info</code> or <code>Debug</code>)
A detailed guide for joining the syslog monitoring scheme is available [[syslog integration guide|here]]
by the sender of the message.


== Goals ==
Configuration allows directing messages to various local devices (console), files (/var/log/) or remote syslog daemons. Care must be taken when updating the configuration as omitting or misdirecting message facilities or severities can cause important messages to be ignored by syslog  or overlooked by the administrator.
* Capture all systems' exception logs.
* Capture selected application error logs.
* Transmit these messages reliably and securely.
* Forward these messages to a central server for storage, analysis and response.
* Track event issues and inform the relevant parties.


== System Architecture ==
==Facility Levels==
=== Logging services ===
{| class="wikitable"
All *nix systems come pre-installed with the BSD distributed syslog(3) logging service. This technology takes messages generated by applications and the kernel and write them to log files located in a configurable but operating system dependent location. These messages may also be forwarded, via the unreliable UDP transport, to a remote host for further processing. This is the system that was in use prior to this syslog project.
|-
! Facility Number !! Facility Description
|-
| 0 || kernel messages
|-
| 1 || user-level messages
|-
| 2 || mail system
|-
| 3 || system daemons
|-
| 4 || security/authorization messages
|-
| 5 || messages generated internally by syslogd
|-
| 6 || line printer subsystem
|-
| 7 ||  network news subsystem
|-
| 8 || UUCP subsystem
|-
| 9 || clock daemon
|-
| 10 || security/authorization messages
|-
| 11 || FTP daemon
|-
| 12 || NTP subsystem
|-
| 13 || log audit
|-
| 14 || log alert
|-
| 15 || clock daemon
|-
| 16 || local use 0  (local0)
|-
| 17 || local use 1  (local1)
|-
| 18 || local use 2  (local2)
|-
| 19 || local use 3  (local3)
|-
| 20 ||  local use 4  (local4)
|-
| 21 || local use 5  (local5)
|-
| 22 || local use 6  (local6)
|-
| 23 || local use 7  (local7)
|}


It was decided that the default syslog service was not flexible or reliable enough to base this project on so a replacement was found.
==Severity levels==


This was the '''syslog-ng''' service, which offers many advantages over the stock BSD syslog service.
{| class="wikitable"
* TCP is used as the transport to ensure reliable message transfer.
|-
* There is a flow-control mechanism to help reliability under heavy load.
! Code !! Severity !! Description !! General Description
* Syslog-ng provided extensive filtering and log rewrite facilities to categorise the incoming messages.
|-
* Syslog-ng understands timezones.
| 0 || Emergency || System is unusable. || A "panic" condition usually affecting multiple apps/servers/sites. At this level it  would usually notify all tech staff on call.
* Syslog can import the log file of any program and convert these entries into syslog messages.
|-
| 1 || Alert || Action must be taken immediately. || Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a backup ISP connection.
|-
| 2 || Critical || Critical conditions. || Should be corrected immediately, but indicates failure in a primary system, an example is a loss of primary ISP connection.
|-
| 3 || Error || Error conditions. || Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
|-
| 4 || Warning || Warning conditions. || Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time.
|-
| 5 || Notice || Normal but significant condition. || Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required.
|-
| 6 || Informational || Informational messages. || Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required.
|-
| 7 || Debug || Debug-level messages. || Info useful to developers for debugging the application, not useful during operations.
|}


=== System roles ===
==Send syslog message to remote host==
Each host within the Infineon organisation will act in one of three roles.
nc -w0 -u 10.151.101.110 514 <<< "<14>User Info msg from remote machine. abjorklund"
# Data collection node
==tcpdump and syslog==
#: This is the most common form of host. It is responsible for capturing messages originating on the host itself. These messages come from two different sources; the syslog(3) logging service and from the monitoring of application log files. These events will be logged to a file, as in the default syslog implementation but will also be forwarded downstream for further processing.
To calculate the priority value the following formula is used : Priority = Facility * 8 + Level
#: There is a Windows service available that allows an NT system to become a source of syslog messages. These messages are not yet fully integrated into the syslog project but the infrastructure is in place to capture and react to Microsoft events.


# Data aggregation node
#!/bin/bash
#: These hosts, one per geographic site, are responsible for receiving all the messages generated by that site and forwarding them downstream for further processing. These may be refered to as '''site hubs''' in other sections of this document.
SYSLOG_PACKAGE=${1}
#: These aggregation nodes are also able to receive message from the old syslog service via the UDP transport. They are therefore able to offer backward compatibility with legacy systems that cannot be modified.
FACILITY=$(echo ${SYSLOG_PACKAGE} / 8 | bc)
#: A DNS alias, local to each site, should be created that names this host '''syslog'''.''site''.'''infineon'''.'''com'''.
SEVERITY_LEVEL=$(echo ${SYSLOG_PACKAGE} - ${FACILITY} \* 8 | bc)
echo Facility: ${FACILITY}
echo Severity level: ${SEVERITY_LEVEL}


# Data analysis node
[root@logserver ~]# tcpdump -Xni eth0 port 514
#: This host is the final destination for all syslog messages within Infineon as a whole or a geographical region, such as Europe. It is here that the data is analysed and an appropriate response is determined. This machine also provides the graphical front-end with which systems administrators interact.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 
  listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
== Data collection node ==
  20:08:05.306002 IP 10.1.1.10.55595 > 10.1.1.1.514: length: 288
Syslog messages are captured by the '''syslog-ng''', or '''syslog''' process on legacy machines, and written to log files. A subset of these messages are then forwarded downstream to the site hub. '''Syslog-ng''' is also able to read application logs and inject their contents into the message stream. The following, possibly incomplete list of applications have been identified as producing logs that should be read and forwarded downstream to the analysis node.
      0x0000: 4500 013c 177d 0000 4011 4b06 0a01 0164  E..<.}..@.K....d
* Samba
      0x0010:  0a01 01c9 d92b a2a2 0128 5f4b 3c31 3431  .....+...(_K<141
* Door Control system
      0x0020:  3e6e 7335 6774 3a20 4e65 7453 6372 6565  >ns5gt:.NetScree
* License daemons
      0x0030: 6e20 6465 7669 6365 5f69 643d 6e73 3567 n.device_id=ns5g
* Clearcase system.
      0x0040: 7420 205b 526f 6f74 5d73 7973 7465 6d2d t..[Root]system-
* LSF
      0x0050: 6e6f
* Apache
=tcp/udp=
 
  @ forward via udp
The location of the log files is operating system dependent under '''syslog'''; on Linux they are placed in /var/log and on Solaris they are placed in /var/adm. '''Syslog-ng''' now always placed them in /var/log, using the Linux standard.
  @@ forward via tcp
 
=logging to individual file=
'''Syslog-ng''' does some local filtering to remove messages that are of no interest to the central servers. For instance it does not forward any cron reports. However all messages are forwarded from the '''syslog''' service so uninteresting messages must be removed by the site hub for legacy systems.
/etc/rsyslog.d/tcp_udp.conf
 
  # Provides UDP syslog reception
If the system to be monitored is Windows based then the '''snare''' service should be installed. The snare server is able to pre-filter the event log messages that it incorporates into the syslog datastream and is also able to apply the required ACLs to the NT filesystem to generate the events that it wishes to capture. For instance, if file modifications to the c:\windows directory are to be monitored then the daemon will apply the correct ACLs to the files in c:\windows. ''It is possible that the system policies editor will interfere with this so this needs some investigation.''
  $ModLoad imudp
 
  $UDPServerRun 514
The local logs are directed to the following files
  # Provides TCP syslog reception
* /var/log/messages
  $ModLoad imtcp
*: This is the default location for all messages that are not directed to one of the other, more specialised, log files.
  $InputTCPServerRun 514
* /var/log/secure
  # bind ruleset to tcp listener
*: This file holds messages referring to authorisation issues. For instance, it captures events associated with incorrect logins, PAM failures and LDAP binding faults, etc.
  $InputTCPServerBindRuleset remote
* /var/log/maillog
  $InputTCPServerRun 514
*: The mail transport service places its output in this file.
  $InputUDPServerBindRuleset remote
* /var/log/cron
  $UDPServerRun 514
*: Each cron job run is logged to this file together with any output that the program or cron generated.
  $template RemoteHost,"/var/log/%HOSTNAME%-%$DAY%.log"
* /var/log/spooler
  # Remote Logging
*: This file will not contain many messages as its contents are produced by the uucp and news system, neither of which are used in this organisation.
  $RuleSet remote
* /var/log/boot.log
  *.* ?RemoteHost
*: Under Linux this file will contain the messages produced during the boot sequence. On a Solaris host it will not contain anything.
Open firewall.
'''Syslog-ng''' is controlled by the /etc/syslog-ng.conf file. An example data-collection node configuration file can be found '''[[syslog-ng.conf-collector|here]]'''.
-A INPUT -p tcp -m state --state NEW -m tcp --dport 514 -j ACCEPT
 
-A INPUT -p udp -m state --state NEW -m udp --dport 514 -j ACCEPT
== Data aggregation node ==
-A OUTPUT -p tcp -m tcp --dport 514 -j ACCEPT
These hosts, there should be one per site, are responsible for collecting all the messages that are generated by the individual '''syslog''' and '''syslog-ng''' processes on that site. The data is then filtered to remove messages that are not of interest and then forwarded downstream for further processing. These hosts will always use the '''syslog-ng''' service for this purpose.
-A OUTPUT -p udp -m udp --dport 514 -j ACCEPT
 
The host chosen for this role should have a DNS alias of '''syslog'''.''site''.'''infineon'''.'''com'''.
 
Note that this node will also contribute to the message stream as it also acts as a data collection node.
 
As '''syslog-ng''' captures authentication problems and can potentially be required to transport confidential information. It is therefore necessary to encrypt this traffic. This [http://www.samag.com/documents/s=9368/sam0506b/0506b.htm paper] was the basis of this work.
 
Using ssh it is possible to build a compressing encrypted tunnel between any two hosts. This can be created by executing the following command on the central server; syslog.intra.infineon.com.
 
  ssh -nNTx -R 8728:localhost:8805 syslog.''site''.infineon.com > /dev/null 2>&1
 
This will make the port syslog.intra.infineon.com:8805 appear as port 8728 on syslog.''site''.infineon.com.
 
If this command is executed as a entry in the inittab of a Solaris/Linux host then the init process will act as a watchdog and will reconnect the tunnel should the connection drop when the machine is at runlevel 3 or 5.
 
log1:35:respawn:/usr/bin/ssh -nNTx -R 8728:localhost:8805 syslog.''site''.infineon.com > /dev/null 2>&1
 
A different port was chosen at each end so that the ssh tunnels could be chained without conflict. At some point in the future even messages internal to a site may be transmitted down these encrypted tunnels.
 
There will be a line in /etc/inittab for each site providing a secured channel to that location.
 
An example /etc/syslog-ng.conf for a Data Aggregation node is provided '''[[syslog-ng.conf-aggregator|here]]'''.
 
[[Image:Syslog servers.jpg|A Data Analysis and Data Agreggation node]]
 
== Data analysis node ==
 
The purpose of the Data analysis node is twofold
 
# Detect and log real-time events, providing an audit log for off-line analysis.
#: The events that this system attempts to capture fall into one of four categories
#* Configuration related events
#* Hardware related events
#* Resource related events
#* Security related events
#: A more detailed list covering these categories can be found [[syslog-event-details|here]].
#: This functionality is provided by the [[http://kodu.neti.ee/~risto/sec/ Simple Event Correlation]] tool.
# Provide an issue tracking mechanism to capture the activities associated with an event.
#: This functionality is provided by an instance of the [[http://bestpractical.com/rt Resource Tracker]] ticketing system.
 
Note that this node will also contribute to the message stream as it also acts as a data collection node
 
=== Message stores ===
In addition to the log files created in /var/log this node will also have a directory called '''/var/log/syslog-ng''' which is populated with a directory named for every host that contributes to the message stream. Within each of these directories are kept files with the name YYYYMMDD. E.g. There will be a file /var/log/syslog-ng/brslcs06/20071009. These files are created at midnight and filled with a day's message stream from that host. The system is configured to hold four weeks data for each host, the last three of which will be compressed via bzip2. These constitute the audit logs and are only consulted manually by humans.
 
The Central Syslog server also writes out a log named '''/var/log/syslog-ng/allmessages''' that contains all the messages from all hosts. This is the source file from which events are extracted.
 
=== Simple Event Correlator(SEC) ===
This software component, which is a perl program, is responsible for reading in events from the '''/var/log/syslog-ng/allmessages''' file and noticing interesting events or sequences of events. It is capable of drawing conclusions from message sequences originating on different hosts and at different times. This is a fairly complex program and its configuration and testing is discussed in more detail '''[[SEC|here]]'''.
 
SEC is located in /opt/sec-2.4.1 on a dedicated Linux node. It reads its configuration files from '''/etc/sec.d''', which contains '''Config.sec.conf''', '''Hardware.sec.conf''', '''Resource.sec.conf''' and '''Security.sec.conf'''. The program runs under the logsurf account and emails its conclusions to the logmon account.
 
The Simple Event Correlator emails any events that it has been configured to notice to an instance of the Resource Tracker Ticketing system. These events will have been allocated to one of the four event categories; configuration, hardware, security and resource and given a priority rating.
 
=== Resource Tracker(RT3) ===
After the data has been collected and analysed the system requires that administrators respond to the events in a timely manner. This is accomplished by emailing the events that SEC has determined are of interest to an [[http://bestpractical.com/rt RT3]] incident monitoring system.  
 
An RT3 system provides
* Authenticated hierarchical logins
* Web based interaction
* Email based interaction. '''Useful for event insertion!'''
* Event tracking throughout its lifetime.
* Automatic event escalation
* Automatic event assignment
* MySQL based histories
* Scripted customised actions
* Structured queries through a web interface.
* PDF Graphic reporting module
 
This product is generally used as a bug tracking system but it is also able to be used for a variety of functions.
 
The RT3 system is configured to read the email generated by the Simple Event Correlator and decode the special subject header to extract the event's type and priority. The site from which the message originated is then used to determine which queue the event should be placed in. There is one queue for each geographical location and this lets administrators focus primarily on their own site. Each administrator who will monitor the events for a site is required to add themselves as a ''watcher'' for their site's queue . This will ensure that they will be emailed with the details of any event that occurs on their site.
 
The RT3 system is installed in /opt/rt3 and is configured by the /opt/rt3/RT_SiteConfig.pm file.
 
  # Any configuration directives you include here will override
# RT's default configuration file, RT_Config.pm
  #
  # To include a directive here, just copy the equivalent statement
# from RT_Config.pm and change the value. We've included a single
  # sample value below.
#
  # This file is actually a perl module, so you can include valid
  # perl code, as well.
  #
# The converse is also true, if this file isn't valid perl, you're
# going to run into trouble. To check your SiteConfig file, use
# this comamnd:
#
#  perl -c /path/to/your/etc/RT_SiteConfig.pm
Set( $rtname, 'infineon.com');
  Set($WebBaseURL , "http://syslog.intra.infineon.com:80");
  Set($WebPath , "");
  ### Change $DatabasePassword to something other than the default "rt_pass"
  ### you don't need to create this user yourself RT will create it while next step
  ### Set($DatabaseUser , "rt_user"); # Only uncomment if you change from the default "rt_user";
  Set($DatabasePassword , "******");
  Set($Organization , "infineon.com");
  Set($RTAddressRegexp , '^logmon\@infineon.com$');
  Set($CanonicalizeEmailAddressMatch , '@brs\.infineon\.com$');
  Set($CanonicalizeEmailAddressReplace , '@infineon.com');
Set($CorrespondAddress , 'logmon');
Set($CommentAddress , 'logmon_Comment');
  Set($MailCommand , 'sendmail');
  Set($SendmailArguments,"-oi -t -ODeliveryMode=b -OErrorMode=m");
  Set($WebExternalAuth , 1);
  Set($WebFallbackToInternalAuth , 1);
  Set($DefaultSummaryRows, 20);
1;
 
The email system must also be updated to forward all email directed at the logmon account into a mail gateway program. This excerpt from the /etc/mail/aliases file shows how this may be done.
 
  logmon: "|/opt/rt3/bin/rt-mailgate --queue Bristol --action correspond --url http://syslog.intra.infineon.com"
  logmon-comment: "|/opt/rt3/bin/rt-mailgate --queue Bristol --action comment --url http://syslog.intra.infineon.com"
 
The site mail administrator is also required to ensure that emails directed to logmon are forwarded to the correct host.
 
Note that the Bristol queue has a special scrip enabled on it that forwards events for other sites to the appropriate queue.
 
Additional details about the customisation of the RT instance are provided '''[[syslog-rt3|here]]'''
 
== The Solaris and Linux build ==
The Solaris and Linux build have been updated to include several packages to support the syslog project. There are also packages that are not installed by default that generate useful syslog output such as Sun's cediag. (This package runs a diagnostic scrub across all DIMMS and reports any anomalies.) These packages should also be added to the client and server builds.<br>
'''Please update this list with any packages that you think should be added to the client builds'''
* '''syslog-ng components for GI4'''
*: [[ftp://ftp.pbone.net/mirror/ftp.silfreed.net/repo/rhel/4/i386/silfreednet/RPMS/eventlog-0.2.5-1.el4.i386.rpm eventlog-0.2.5-1.el4.i386.rpm ]]
*: [[ftp://ftp.pbone.net/mirror/ftp.silfreed.net/repo/rhel/4/i386/silfreednet/RPMS/syslog-ng-2.0.4-1.el4.i386.rpm syslog-ng-2.0.4-1.el4.i386.rpm ]]
* '''syslog-ng components for GI3'''
*: [[ftp://ftp.silfreed.net/repo/rhel/3/i386/silfreednet/RPMS/eventlog-devel-0.2.5-1.el3.i386.rpm eventlog-0.2.5-1.el3.i386.rpm ]]
*: [[ftp://ftp.silfreed.net/repo/rhel/3/i386/silfreednet/RPMS/syslog-ng-2.0.4-1.el3.i386.rpm syslog-ng-2.0.4-1.el3.i386.rpm ]]
* cediag for Solaris
 
=== Solaris packaging ===
An attempt was made to provide the Solaris syslog-ng package as statically linked binary but this was not possible due to the following reasons.
The Solaris version of the syslog-ng usss internationalization support that looks up strings in a text library. The binary therefore require support services which cannot be linked statically so the Solaris pkg will have to contain this text database. As this makes the package contain a binary and a text library lookup area it would make little sense to link the whole program statically. The final package holds several libraries but stores them in a single location so that it does not interact with /opt/TWWfsw and other library areas after the system has gone multi-user and mounted other packages.
It installs into '''/opt/syslog-ng_2.0.5''', which contains .../bin, .../share, .../lib, etc
 
The Solaris package file can be found [http://brscs09.brs.infineon.com/splat/stable/sparc/5.8/syslogng-2.0.5-SunOS5.8-all-IFX.pkg here]
 
Remember to update /etc/syslog-ng.conf with one of these, based upon the host's role.
* http://svn.klu.infineon.com/repos/AdminToolKit/trunk/conf/push/standard/etc/syslog-ng.conf.host.Solaris
* http://svn.klu.infineon.com/repos/AdminToolKit/trunk/conf/push/standard/etc/syslog-ng.conf.hub.Solaris
 
=== Linux packaging ===
RedHat 3 requires these two packages to support syslog-ng
* [[ftp://ftp.silfreed.net/repo/rhel/3/i386/silfreednet/RPMS/eventlog-devel-0.2.5-1.el3.i386.rpm eventlog-0.2.5-1.el3.i386.rpm ]]
* [[ftp://ftp.silfreed.net/repo/rhel/3/i386/silfreednet/RPMS/syslog-ng-2.0.4-1.el3.i386.rpm syslog-ng-2.0.4-1.el3.i386.rpm ]]
 
RedHat 4 requires these two packages to support syslog-ng
* [[ftp://ftp.pbone.net/mirror/ftp.silfreed.net/repo/rhel/4/i386/silfreednet/RPMS/eventlog-0.2.5-1.el4.i386.rpm eventlog-0.2.5-1.el4.i386.rpm]]
* [[ftp://ftp.pbone.net/mirror/ftp.silfreed.net/repo/rhel/4/i386/silfreednet/RPMS/syslog-ng-2.0.4-1.el4.i386.rpm syslog-ng-2.0.4-1.el4.i386.rpm]]
RedHat 2 packages to follow.
 
The configuration files must then be updated manually or via puppet from the [[http://svn.klu.infineon.com/wsvn/AdminToolKit/trunk/conf/push/standard/etc svn repository]]. The three type are central, (for the master syslog server,) site, (for the host responsible for forwarding syslog streams from a site,) and host, (a simple drop-in replacement for the syslog service.)
 
You will probably need http://svn.klu.infineon.com/repos/AdminToolKit/trunk/conf/push/standard/etc/syslog-ng.conf.host.RedHat. Also remember to '''chkconfig --levels 2345 syslog off''' if installing manually.
 
==Central Server System Specification==
Using the experience gained during this investigation the specification for the target system should be
* Quad core AMD64 hardware.
*: Highest Performance is not required as the number of cores is more important. The software system has a large grain parallelism that would allow us to run the computationally intensive SEC processes in parallel. By default sec reads all four configuration classes and then processes the '''allmessages''' file to generate its event messages. It should be possible to run four instances of sec each only processing one of the event class configuration files. However, it may be that some cores should be left free to process the RT requests via Apache. Tuning will be an ongoing task.
* 3GB+ memory.
*: The SEC processes have a high memory requirement as they have to keep an in-core image of all the outstanding events that are being correlated. I have observed a peak memory size of 75MB for ~60 hosts this would lead to ~1.2GB for 1000 hosts. Allowing for other processes on the server and rule database extensions a value of 3GB+ should be adequate. However the DIMM sockets should not be fully populated to allow for errors in this calculation.
* Twin drives.
*: The system should be configured with a level-1 raid drive. There is no need for particularly fast disk IO on this system as the data rate is not very high, just continuous. The raid is just to provide high availability. Given current disk prices the system should provide 72GB of user space(2*72GB)
* RedHat Enterprise Linux 4.
*: A GoldImage 4 machine offers all the features needed to support SEC and RT3. Note that Solaris would be able to provide the required features but is much more difficult to configure.
* Software requirements beyond a standard GI4 client
** sec-2.4.1-1.0.el4.ifx.noarch.rpm
*** configured by puppet with files in svn
** rt-3.6.4-1.0.el4.ifx.x86_64.rpm
*** configured by puppet with files in svn
** Required RHEL4 packages
*** httpd-devel
*** apr-devel
*** mysql
*** mysqlclient10
*** mysql-server
*** mysql-devel
*** gcc
*** sendmail-cf
*** mod_perl
*** freetype-devel
*** gd-devel
*** libjpeg-devel
*** libpng-devel
*** xorg-x11-devel
*** perl-* (yes, all the perl modules)
** mod_fastcgi_2.4.2-1.0.el4.ifx.x86_64.rpm
** dag.wieers.com packages. This revision or newer.
*** perl-Apache-Session-1.80-1.2.el4.rf.noarch.rpm
*** perl-Apache-Session-1.81-1.el4.rf.noarch.rpm
*** perl-Archive-Tar-1.32-1.el4.rf.noarch.rpm
*** perl-Cache-Simple-TimedExpiry-0.27-1.el4.rf.noarch.rpm
*** perl-Calendar-Simple-1.17-1.el4.rf.noarch.rpm
*** perl-Class-Container-0.12-1.2.el4.rf.noarch.rpm
*** perl-Class-Data-Inheritable-0.06-1.el4.rf.noarch.rpm
*** perl-Class-ReturnValue-0.53-1.2.el4.rf.noarch.rpm
*** perl-Class-Singleton-1.03-1.2.el4.rf.x86_64.rpm
*** perl-Clone-0.22-1.el4.rf.x86_64.rpm
*** perl-Convert-BinHex-1.119-2.2.el4.rf.noarch.rpm
*** perl-Crypt-DES-2.05-3.2.el4.rf.x86_64.rpm
*** perl-Crypt-SSLeay-0.56-1.el4.rf.x86_64.rpm
*** perl-DateTime-0.2901-1.2.el4.rf.x86_64.rpm
*** perl-DateTime-Format-Mail-0.30-1.el4.rf.noarch.rpm
*** perl-DateTime-Format-W3CDTF-0.04-1.el4.rf.noarch.rpm
*** perl-DateTime-Locale-0.22-1.2.el4.rf.x86_64.rpm
*** perl-DateTime-TimeZone-0.46-1.el4.rf.x86_64.rpm
*** perl-DBI-1.58-2.el4.rf.x86_64.rpm
*** perl-DBIx-DBSchema-0.31-1.el4.rf.noarch.rpm
*** perl-DBIx-SearchBuilder-1.49-1.el4.rf.noarch.rpm
*** perl-Devel-StackTrace-1.15-1.el4.rf.noarch.rpm
*** perl-Exception-Class-1.23-1.el4.rf.noarch.rpm
*** perl-FCGI-0.67-1.2.el4.rf.x86_64.rpm
*** perl-GD-2.35-1.el4.rf.x86_64.rpm
*** perl-GD-Graph-1.43-0.2.el4.rf.x86_64.rpm
*** perl-GD-Graph-1.43-1.2.el4.rf.noarch.rpm
*** perl-GD-Text-Util-0.86-0.2.el4.rf.x86_64.rpm
*** perl-GD-Text-Util-0.86-1.2.el4.rf.noarch.rpm
*** perl-Hook-LexWrap-0.20-1.el4.rf.noarch.rpm
*** perl-HTML-Format-2.04-1.2.el4.rf.noarch.rpm
*** perl-HTML-Mason-1.3200-1.el4.rf.noarch.rpm
*** perl-HTML-Parser-3.55-1.el4.rf.x86_64.rpm
*** perl-HTML-Scrubber-0.08-1.2.el4.rf.noarch.rpm
*** perl-HTML-Tagset-3.10-1.el4.rf.noarch.rpm
*** perl-HTML-Tree-3.23-1.el4.rf.noarch.rpm
*** perl-HTTP-Server-Simple-0.27-1.el4.rf.noarch.rpm
*** perl-HTTP-Server-Simple-Mason-0.09-1.el4.rf.noarch.rpm
*** perl-IO-Socket-INET6-2.51-1.2.el4.rf.noarch.rpm
*** perl-IO-Socket-SSL-1.07-2.el4.rf.noarch.rpm
*** perl-IO-stringy-2.110-1.2.el4.rf.noarch.rpm
*** perl-IO-Zlib-1.05-1.el4.rf.noarch.rpm
*** perl-LDAP-0.34-1.el4.rf.noarch.rpm
*** perl-Locale-Maketext-Fuzzy-0.02-1.2.el4.rf.noarch.rpm
*** perl-Locale-Maketext-Lexicon-0.64-1.el4.rf.noarch.rpm
*** perl-Log-Dispatch-2.18-1.el4.rf.noarch.rpm
*** perl-Mail-Sender-0.8.13-1.el4.rf.noarch.rpm
*** perl-Mail-Sendmail-0.79-1.2.el4.rf.noarch.rpm
*** perl-MailTools-1.77-1.el4.rf.noarch.rpm
*** perl-MIME-Lite-3.01-2.2.el4.rf.noarch.rpm
*** perl-MIME-tools-5.420-2.el4.rf.noarch.rpm
*** perl-Module-Versions-Report-1.03-1.el4.rf.noarch.rpm
*** perl-Net-Daemon-0.43-1.el4.rf.noarch.rpm
*** perl-Net-DNS-0.61-1.el4.rf.x86_64.rpm
*** perl-Net-IP-1.25-1.el4.rf.noarch.rpm
*** perl-Net-SNMP-5.0.1-1.2.el4.rf.noarch.rpm
*** perl-Net-SNMP-5.2.0-1.2.el4.rf.noarch.rpm
*** perl-Net-SSLeay-1.30-1.el4.rf.x86_64.rpm
*** perl-Params-Validate-0.86-1.el4.rf.x86_64.rpm
*** perl-PlRPC-0.2020-1.el4.rf.noarch.rpm
*** perl-Regexp-Common-2.120-1.2.el4.rf.noarch.rpm
*** perl-Socket6-0.19-1.2.el4.rf.x86_64.rpm
*** perl-Term-ReadKey-2.30-2.el4.rf.x86_64.rpm
*** perl-Text-Autoformat-1.13-1.2.el4.rf.noarch.rpm
*** perl-Text-Quoted-2.02-1.el4.rf.noarch.rpm
*** perl-Text-Reform-1.11-1.2.el4.rf.noarch.rpm
*** perl-Text-Template-1.44-1.2.el4.rf.noarch.rpm
*** perl-Text-WikiFormat-0.78-1.el4.rf.noarch.rpm
*** perl-Text-Wrapper-1.000-1.2.el4.rf.noarch.rpm
*** perl-TimeDate-1.16-1.2.el4.rf.noarch.rpm
*** perl-Time-modules-2003.1126-1.2.el4.rf.noarch.rpm
*** perl-Tree-Simple-1.17-1.el4.rf.noarch.rpm
*** perl-UNIVERSAL-require-0.11-1.el4.rf.noarch.rpm
*** perl-Want-0.14-1.el4.rf.x86_64.rpm
*** perl-XML-Dumper-0.81-1.el4.rf.noarch.rpm
*** perl-XML-NamespaceSupport-1.09-1.2.el4.rf.noarch.rpm
*** perl-XML-RSS-1.22-1.el4.rf.noarch.rpm
*** perl-XML-SAX-0.16-1.el4.rf.noarch.rpm
*** perl-XML-Twig-3.29-1.el4.rf.noarch.rpm
*** python-urlgrabber-2.9.7-1.2.el4.rf.noarch.rpm
*** spamassassin-3.2.3-1.el4.rf.x86_64.rpm
 
These packages have been added to a yumgroup called '''Central syslog server + RT3'''.
 
== Central Server Deployment ==
Hardware has been allocated at the Klagenfurt site for the deployment of the European Central Server, syslog.intra.infineon.com. It will be hosted by a virtual machine running under VMware and will be initially provisioned with 4 cores and 10GB of SAN disk.
== Post Install Operations ==
The system can be built using puppet however some additional configuration was required to customize the RT instance for Infineon Event tracking.
* Removed or edited all global scrips that replied to requestor
*: These can be found at Configuration->Global->Scrips
*: The requestor will always be user logsurf. No Emails should be sent to this account.
* Changed the "real name" for user root from "Enoch Root" to "Root"
*: This can be found at Configuration->Users->root
* Created a Global Ticket Custom Field called "Event Type"
*: This can be found at Configuration->Global->Custom Fields->Ticket
*: When an event is generated by SEC it is placed into one of four categories.
** Configuration issues
** Hardware issues
** Resource issues
** Security issues
*: The custom field is a Combo box type with a name and description matching the event types
* Created a Global Ticket Custom Field called "Host"
*: This can be found at Configuration->Global-<Custom Fields->Ticket
*: This field holds the name of the host on which the event originated.
*: The custom field type is "Enter one value"
* The '''General''' queue was renamed '''Bristol'''
*: This can be found at Configuration->Queues->General
* A queue was created for all other participating sites.
*: This can be found at Configuration->Queues->New Queue
* Two On-Create scrips were applied to the Bristol queue
*: This can be found at Configuration->Queues->Bristol->Scrips->New Scrip
*: '''Scrip one'''
** Description: Set Custom Field From Subject
** Condition: On Create
** Action: Set Custom Fields By Subject
** Template: Global Template: Blank
** Stage: TransactionCreate
*: '''Scrip two'''
** Description: Set Queue From Subject Field
** Condition: On Create
** Action: Set Queue By Subject
** Template: Global Template: Blank
** Stage: TransactionCreate
* A user account was created for all interested parties
*: This can be found at Configuration->Users->New User
* A group account was created for each site or queue called ''<site>'''''-it'''
*: This can be found at Configuration->Groups->New Group
* A user that wishes to be alerted to events on the Bristol queue would place themselves in the '''Bristol-it''' group
*: This can be found at Configuration->Groups->Bristol-it->Members
* A group called '''it-access''' was created into which all the users were placed.
*: This can be found at Configuration->Groups->New Group and Configuration->Groups->it-access->Members
* This group was granted system rights to access and modify tickets.
*: This can be found at Configuration->Global->Group Rights
 
== Future Improvements ==
 
== Initial project Investigation ==
The inital project investigation plan can be found [[syslog-investigation|here]]
 
== Notes ==
=== Message Matrix ===
The matrix is very difficult to maintain in Wiki format so I have placed it in this [http://ishare.infineon.com/personal/pkelly/syslog/Shared%20Documents/Forms/AllItems.aspx sharepoint]
 
=== Severity levels ===
<small><pre>LOG_EMERG
      A panic condition.
 
LOG_ALERT
      A condition that should be corrected immediately, such as a corrupted system database.
 
LOG_CRIT
      Critical conditions, such as hard device errors.
 
LOG_ERR
      Errors.
 
LOG_WARNING
      Warning messages.
 
LOG_NOTICE
      Conditions that are not error conditions, but that may require special handling.
 
LOG_INFO
      Informational messages.
 
LOG_DEBUG
      Messages that contain information normally of use only when debugging a program.
</pre></small>
 
== Meetings ==
 
[[syslog 8/8/2007]]
 
[[syslog 16/8/2007]]
 
[[sylog 19/9/2007]]
[[Category:Applications]]
[[Category:Unix]]

Latest revision as of 16:34, 14 September 2020

syslog

Messages refer to a facility (auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0, ... , local7 ) and are assigned a severity (Emergency, Alert, Critical, Error, Warning, Notice, Info or Debug) by the sender of the message.

Configuration allows directing messages to various local devices (console), files (/var/log/) or remote syslog daemons. Care must be taken when updating the configuration as omitting or misdirecting message facilities or severities can cause important messages to be ignored by syslog or overlooked by the administrator.

Facility Levels

Facility Number Facility Description
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon
10 security/authorization messages
11 FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 clock daemon
16 local use 0 (local0)
17 local use 1 (local1)
18 local use 2 (local2)
19 local use 3 (local3)
20 local use 4 (local4)
21 local use 5 (local5)
22 local use 6 (local6)
23 local use 7 (local7)

Severity levels

Code Severity Description General Description
0 Emergency System is unusable. A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call.
1 Alert Action must be taken immediately. Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a backup ISP connection.
2 Critical Critical conditions. Should be corrected immediately, but indicates failure in a primary system, an example is a loss of primary ISP connection.
3 Error Error conditions. Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
4 Warning Warning conditions. Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time.
5 Notice Normal but significant condition. Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required.
6 Informational Informational messages. Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required.
7 Debug Debug-level messages. Info useful to developers for debugging the application, not useful during operations.

Send syslog message to remote host

nc -w0 -u 10.151.101.110 514 <<< "<14>User Info msg from remote machine. abjorklund"

tcpdump and syslog

To calculate the priority value the following formula is used : Priority = Facility * 8 + Level 

#!/bin/bash
SYSLOG_PACKAGE=${1}
FACILITY=$(echo ${SYSLOG_PACKAGE} / 8 | bc)
SEVERITY_LEVEL=$(echo ${SYSLOG_PACKAGE} - ${FACILITY} \* 8 | bc)
echo Facility: ${FACILITY}
echo Severity level: ${SEVERITY_LEVEL}

[root@logserver ~]# tcpdump -Xni eth0 port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:08:05.306002 IP 10.1.1.10.55595 > 10.1.1.1.514: length: 288
      0x0000:  4500 013c 177d 0000 4011 4b06 0a01 0164  E..<.}..@.K....d
      0x0010:  0a01 01c9 d92b a2a2 0128 5f4b 3c31 3431  .....+...(_K<141
      0x0020:  3e6e 7335 6774 3a20 4e65 7453 6372 6565  >ns5gt:.NetScree
      0x0030:  6e20 6465 7669 6365 5f69 643d 6e73 3567  n.device_id=ns5g
      0x0040:  7420 205b 526f 6f74 5d73 7973 7465 6d2d  t..[Root]system-
      0x0050:  6e6f

tcp/udp

@  forward via udp
@@ forward via tcp

logging to individual file

/etc/rsyslog.d/tcp_udp.conf 

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$template RemoteHost,"/var/log/%HOSTNAME%-%$DAY%.log"
# Remote Logging
$RuleSet remote
*.* ?RemoteHost

Open firewall. 

-A INPUT -p tcp -m state --state NEW -m tcp --dport 514 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 514 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 514 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 514 -j ACCEPT
Retrieved from "https://halfface.se/wiki/index.php?title=Syslog&oldid=13486"