Attempting to configure syslog-ng on AWS Linux 2 EC2

[e39_540i]

Hi all,

As the title suggests, I'm attempting to configure syslog-ng on an AWS Linux 2 EC2 instance and I should state that I'm quite a novice when it comes to Linux so there were some stumbling blocks getting the syslog server setup but I believe I've got everything running now, unfortunately, I'm running into some error messages when I attempt to configure one of our appliances, a virtualized VMX100, in the same AWS environment, to send its logs to the server.

I confirmed with Meraki support that they could see traffic being sent out via port 514. However, when I configure the s_net source in the syslog-ng.conf file, as soon as I switch the IP address or uncomment the line and enter the IP address of the VMX, the service fails shortly after. Or, if I restart the service, it will fail to start. There seems to be an issue binding the IP address and port. As soon as I comment out the line and go back to the mostly default config, the syslog-ng service runs fine without issues.

Attached is a screenshot of the exact error message I get whenever I try to switch that source to the IP of the VMX100"

some more info:

journalctl -xe

- The result is failed.

Jan 22 18:58:37 ip-addr-here systemd[1]: Unit syslog.socket entered failed state.

Jan 22 18:58:37 ip-addr-here systemd[1]: syslog-ng.service failed.

Jan 22 18:58:37 ip-addr-here dhclient[2880]: XMT: Solicit on eth0, interval 130400ms.

----

Jan 22 18:51:52 ip-ip-addr-here syslog-ng[8939]: syslog-ng starting up; version='3.5.6'
Jan 22 18:51:52 ip-ip-addr-here systemd[1]: Listening on Syslog Socket.
Jan 22 18:51:52 ip-ip-addr-here systemd[1]: Starting Syslog Socket. 
Jan 22 18:51:52 ip-ip-addr-here systemd[1]: Starting System Logger Daemon...
Jan 22 18:51:52 ip-ip-addr-here systemd[1]: Started System Logger Daemon.
Jan 22 18:52:39 ip-ip-addr-here dhclient[2880]: XMT: Solicit on eth0, interval 128120ms.
Jan 22 18:54:48 ip-ip-addr-here dhclient[2880]: XMT: Solicit on eth0, interval 108460ms.
Jan 22 18:56:36 ip-ip-addr-here dhclient[2880]: XMT: Solicit on eth0, interval 120920ms.
Jan 22 18:56:50 ip-ip-addr-here syslog-ng[8939]: syslog-ng shutting down; version='3.5.6'
Jan 22 18:56:50 ip-ip-addr-here systemd[1]: Stopping System Logger Daemon...
Jan 22 18:56:50 ip-ip-addr-here syslog-ng[8952]: syslog-ng starting up; version='3.5.6'
Jan 22 18:56:50 ip-ip-addr-here systemd[1]: Stopped System Logger Daemon.
Jan 22 18:56:50 ip-ip-addr-here systemd[1]: Starting System Logger Daemon...
Jan 22 18:56:50 ip-ip-addr-here systemd[1]: Started System Logger Daemon.
Jan 22 18:58:00 ip-ip-addr-here syslog-ng[8952]: syslog-ng shutting down; version='3.5.6'
Jan 22 18:58:00 ip-ip-addr-here systemd[1]: Stopping System Logger Daemon...

Any help would be greatly appreciated!

[PhilipDAth]

I don't know the answer.

I believe the s_net is the interface on the syslog service that will be used to listen to incoming messages - not the IP address of the VMX.

Try:

source s_net { tcp(ip(0.0.0.0) port(514) max-connections (5000)); udp(); };

[e39_540i]

Thanks for your reply, @PhilipDAth. Based on the documentation here:

https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Syslog_Server_Overv...

Is there any issue with that initial part of the statement that shows "tcp" since the document linked above shows that Meraki sends syslog over UDP?

I'll give this a shot.

[CptnCrnch]

Correct, Meraki uses standard Syslog on 514/UDP.

[e39_540i]

---

@PhilipDAth wrote:

I don't know the answer.

I believe the s_net is the interface on the syslog service that will be used to listen to incoming messages - not the IP address of the VMX.

Try:

source s_net { tcp(ip(0.0.0.0) port(514) max-connections (5000)); udp(); };

---

That was totally my fault. I had misread that particular line in the documentation and entered the source s_net as the IP address of the VMX. 

I've corrected that and the service is no longer failing with the IP address of the syslog service. However, I'm not seeing it listening. Any ideas if I need to manually set it to listen or should it be listening by default once configured?

[ww]

Last time i setup  syslog on Linux had similar issue and i ended up changing the port to 1514 an then it worked fine. 

I believe you can only run on port 514 if you run the programm  as root 

[PhilipDAth]

Install tcpdump, and try some commands like:

tcpdump src <ip of VMX>
tcpdump -n udp

Are you seeing the syslog traffic?  If not, is the AWS security group allowing the traffic.  Can you ping the VMX?

If you are seeing the traffic, then it is a local configuration issue on the Linux box. 

[e39_540i]

Thanks for your help! Looks like I had misconfigured the security group and set it for TCP instead of UDP. I configured my other firewall correctly for UDP but wasn't testing with that device just yet. Recreated the security group and I'm now receiving data to my flows and urls log files! BTW, I am still using port 514 and did not have to switch to 1514 (will keep this in mind, though, if we run into issues in the future).

I'm now working on trying to get these log files ingested by a SIEM solution (Sumo Logic) we're looking to implement so this has been a really good learning lesson for me. I may continue to post here rather than open up a new thread just to keep things organized in case anyone else runs into issues configuring syslog-ng on an AWS Linux 2 EC2 instance.

[Sumo_int9898]

Hello, I am currently working on getting a syslog server installed in an EC2 instance. My questions is where are you directing the logs from the sources into an S3 bucket?

[e39_540i]

Hey @Sumo_int9898 , I would have loved to have pushed it to an S3 bucket but they are going to the EC2 instance itself. Once Sumo ingests the log, the data in the log file itself is no longer required. You should also be able to set a retention policy on the Sumo Logic side for compliance purposes. You could theoretically set a cron job on the EC2 instance to clear out the files on a daily basis as they would not be needed for any considerable period.

[Sumo_int9898]

Thanks for the information thats helpful, I might post here in future if I have any questions related to  Sumo Logic and Syslog server.

[e39_540i]

Hey just a heads up, I went back and cleaned this up and removed syslog-ng altogether as I didn't want to have to maintain a different application just to collect the Meraki logs. I might re-install it if I need to push other logs to this server but for the time being, I'm only sending Cisco FP and Meraki logs. 

In order to do this, I downloaded and installed the Sumo Logic Installed Collector on my Linux server. With the collector installed, you simply need to configure the sources (your Meraki devices) in the Sumo Logic dashboard with the corresponding IP and port numbers.

Use these docs to get things going:

1. https://help.sumologic.com/07Sumo-Logic-Apps/22Security_and_Threat_Detection/Cisco_Meraki/Collect_lo...

2. https://help.sumologic.com/03Send-Data/Sources/01Sources-for-Installed-Collectors/Syslog-Source

From this article, this was the important piece for me -- 

For multiple syslog collections, set up a separate Source for each and set a separate port number for each.

Hope that helps. I'm now able to ingest the Meraki logs without configuring a whole bunch of overhead for syslog-ng.