Getting started with Splunk

Splunk lets you index, search, alert and report on any IT data, in real time, for application management, IT operations, security and compliance, and more. Splunk consumes any data: log files, system metrics, applications, configurations...feed any data you want into Splunk.

Click on a topic below to get started.

Getting around

Splunk's UI is packed with features. Read through the following topics to get a better sense of how to navigate and use your Splunk installation.

Where am I right now?

Splunk is made up of apps. Apps create different contexts for your data out of sets of views, dashboards, and configurations. Right now, you're in the Getting Started app. You also have the search app, which is where you can create searches, and the default app Splunk Home, which lets you launch other apps. You can also add new apps from Splunkbase, or create your own.

Navigate between apps

To navigate to another app, use the App drop-down in the upper right hand corner:

screenshow_uploadLocal

To see a list of apps that are currently installed in your Splunk instance, you can return to Home by clicking the App menu in the upper right hand corner of this page and choosing Home. This will take you out of the Getting Started app, but you can get back here by choosing Getting Started again from the Home or the App menu.

Used Splunk before? Looking for something a little more familiar?

If you've used Splunk before, you're probably looking for the Search app. To get to the Search app, choose "Search" from the App menu in the upper right-hand corner.

Manage your Splunk install

Most of Splunk's management options are available through Splunk Web (Splunk's user interface). Note that some configurations are only available to Splunk users with admin privileges. If you can't access some of the configurations discussed in this app, you may not have permission to access them.

Use Splunk Manager

Manage your configurations and apps with Splunk Manager. Almost every configuration change can be set through Splunk Manager. Get to Manager by clicking on the Manager link in the upper right hand corner:

screenshow_uploadLocal

Use Job Manager

Manage your searches with Job Manager. All of your searches run as jobs. You can list and control all the searches running on your system by clicking the Jobs link in the upper right hand corner:

screenshow_uploadLocal

Add more apps

To browse for and download more apps for Splunk, return Home and click the Find more apps button.

You can make your own apps, too. Refer to the Developer Manual for more information.

Index data

If a machine generates it, Splunk can index it. Yep, that's right -- Splunk can index any data, structured or unstructured, without custom parsers, connectors, or adapters. Feed Splunk anything from syslog from Unix servers and network devices, to Event Logs on Windows, to custom application logs and even configurations and system metrics -- you'll get visibility into your entire operation.

You can use Splunk's getting data in workflows to find out all the different types of data you can add to Splunk, and your different options for getting this type of data in. There are lots of ways to get your data into Splunk, depending on where it's located.

On this Splunk server

Splunk can index any local data, either continuously or just once. Configure Splunk to index a file or directory through Splunk Manager. You can also preview your data before you index it and create custom rules for how Splunk should handle your data.

  1. Click Manager » Data inputs » Files & Directories.
  2. Click New.
  3. This will take you to Data Preview and let you create a sourcetype -- rules for how Splunk should parse and extract index time data from your files and directories.
  4. Once you've finished previewing your data, you can add a single file or a directory and Splunk will index the data immediately.
  5. Choose ''Continuously index data from a file or directory this Splunk instance can access'' to continuously index data.
  6. Choose ''Upload and index a file'' to upload a file on your client machine.
  7. Choose ''Index a file once from this Splunk server'' to index a file once.
  8. Set more options by following the rest of the directions on the page. For example, set a host name, sourcetype or index. Or you can leave most of these options to the defaults and Splunk will assign them for you.

On a remote Windows machine

Use Splunk to gather Windows data from other machines in your network. You have a few options, depending on how your network is set up:

  1. If you're running Splunk on Windows, you can use WMI to collect Windows event logs, as well as Event Log collection, registry monitoring and Active Directory monitoring. All your options are listed in Manager » Data inputs.
  2. If you're running Splunk on a non-Windows machine, you can use an agent to send the data from your Windows machine to Splunk. Install a universal forwarder on the remote machine. For more information about forwarders, check out the Universal forwarder topic in the documentation.
  3. Or, you can always index data from directories that are shared on your network. Just navigate to Splunk Manager Data Inputs page and choose ''Monitor a file or directory''. Type in the full path to the file or directory. For example, on Windows you can enter: c:\apache\apache.error.log to monitor a local file, or \\hostname\apache\apache.error.log to monitor a file on a remote Windows machine.

On a remote non-Windows machine

Use Splunk to gather data from other non-Windows machines in your network. You have a few options, depending on how your network is set up:

  1. You can always index data from directories that are shared on your network. Just navigate to Splunk Manager Data Inputs page and choose ''Monitor a file or directory''. Type in the full path to the file or directory. On Unix, use the form /var/log to monitor a local file, or /mnt/www01/var/log to monitor a remote directory.
  2. You can also use Splunk's universal forwarder to send the data from any other machine to Splunk. Install the forwarder as an agent on the remote machine, and set it up to send data to your Splunk server. For more information about forwarders, check out the Universal forwarder topic in the documentation.

On your network

Use this method to capture data sent over a TCP or UDP port. For example, set up Splunk to listen on UDP 514 to capture syslog data.

  1. Navigate to the Data Inputs page within Splunk Manager.
  2. Click New next to UDP to add data from UDP port.
  3. Click New next to TCP to add data from a port using TCP.
  4. Specify the port Splunk should listen on.
  5. Set more options by following the rest of the directions on the page.

More input methods

There are other ways to get your data into Splunk. Here are a few popular options:

Collect data from several machines in a distributed environment

Wonder how you'll get data to Splunk from a distributed environment, such as a farm of app or web servers logging locally? Splunk's universal forwarder is a lightweight agent that can be deployed to dozens or even hundreds or thousands of servers to capture data in real time and send to a central Splunk indexer. Use Universal forwarders to send data to Splunk from other systems. Set up Forwarding from Splunk Manager.

Script your own inputs

Create a scripted input for your custom data source. Scripted input are useful for command-line tools such as vmstat, iostat, netstat, top, etc. Get data from APIs and other remote data interfaces and message queues and generate metrics and status data from exercising system and app status commands like vmstat, iostat, etc. Lots of apps on SplunkBase provide scripted inputs for specific applications. Set up scripted inputs from Splunk Manager.

Monitor file system changes

Interested in what changes are happening on your file system? Set up file system change monitoring and see every change as it occurs. Use this method to monitor critical files, configuration files, and more as required for many compliance mandates as well as to find system-impacting changes and unauthorized changes for security and operations.

Search

Once you have data in Splunk, you can use the Search app to investigate security incidents, troubleshoot application, server and network problems, or just proactively review system and user activity.

Free form search

Search for any text that you expect to find in your data.

  1. Navigate to the Search app.
  2. Type terms directly into the search bar. If you are investigating a problem, search for:
  3. screenshow_uploadLocal
  4. Combine terms with Boolean expressions. So, if you want to find errors that are not associated with Web activity, search for:
  5. screenshow_uploadLocal
  6. Use wildcards to match patterns of terms. If you want to find failed login attempts, which could include both "failed" or "failure" in the message, you might search for:
  7. screenshow_uploadLocal

Search in real time

Search data in real time as it comes into Splunk.

  1. Navigate to the Search app.
  2. Type in a free-form search. Real-time search supports all of Splunk's search language.
  3. Select the ''Real-time'' option, and then pick a window of time over which to display your real-time results as they stream in.
  4. screenshow_uploadLocal

Interact with results

Your search results are just as interactive as the timeline. In this section, you'll see how, with just one click, you can add, remove, and exclude terms from your search.

  1. Navigate to the Search app.
  2. Run a search for any term in your data.
  3. Move your mouse over your search results. Notice that words and phrases highlight as you mouse over them, indicating that you can add these terms to your search.
  4. Highlight and click a term in your search results. Your search updates to include this term in the search bar and filters out all previous results that don't match.
  5. Alternately, click any term in your search results that is already highlighted; Splunk updates and removes that term from your search.
  6. Additionally, you can specify terms for Splunk to exclude. Highlight the term and alt-click (in Windows, ctrl-click); Splunk updates your search to exclude this term with a Boolean NOT operation.

Use fields to search

Free form search is easy and powerful, but it doesn't always give you the answer that you want. For example, you may want to exclude events with the HTTP status code 200. But, if you just search for "NOT 200", you'll also remove events you might want to keep, such as "503" status events coming from IP addresses with 200 in them.

As Splunk indexes every term in your original data, it discovers and adds fields based on name/value pairs, headers, or other information that is otherwise self-explanatory. For example, Splunk automatically adds information about where the data came from into host, source and sourcetype fields. Splunk might also recognize other parts of your data, such as IP addresses, HTTP status codes, etc. You can also add your own fields, as discussed in the Add knowledge section of this app.

  1. Navigate to the Search app.
  2. For example, search for web activity:
  3. screenshow_uploadLocal
  4. Notice the Fields menu next to the search results, on the left hand side. These are fields that Splunk automatically discovers and adds.
  5. screenshow_uploadLocal

    Fields that are visible in your search results are listed under the 'selected fields' header. You can select more fields to show. Other fields that Splunk discovered automatically are listed under 'interesting fields'.

  6. Next to each field name is the number of different values that exist for the field in your search results. Click on any of the field names to see the top values of each field. Click on any of the field values to add it as a filter to your search.
  7. .
  8. If you are searching through web data, you can add the HTTP status to the fields menu by clicking 'Pick fields' and picking 'status' from the pop-up menu that appears.
  9. Notice that the values of the 'status' field are the HTTP status codes: 200, 503, 404, etc. Now, you can use this knowledge to search for or exclude specific field values. A search for all unsuccessful Web access events might be:
  10. screenshow_uploadLocal
  11. You can also use comparison operators ( >, <, >=, <=) when searching with fields; to see all events with status values greater than 300, search for:
  12. screenshow_uploadLocal

Investigate with the timeline

The timeline is a visual representation of the number of events that occur at each point in time. Thus, you can use the timeline to highlight patterns of events or investigate peaks and lows in event activity.

  1. Navigate to the Search app.
  2. Try running a search for 'error' and notice the timeline right below your search.
  3. screenshow_uploadLocal
  4. As the timeline updates with your search results, you might notice clusters or patterns of events; peaks or valleys in the timeline can indicate spikes in activity or server downtime.
  5. Click on a point in the timeline and drag your mouse over a cluster of bars to a second point. Your search results update to display only the events that occurred in that selected time range.
  6. screenshow_uploadLocal
  7. Click on one bar in the timeline. Your search results update to display only the events that occur at that selected point in time.
  8. screenshow_uploadLocal

Use search assistant

Search assistant is a quick in-product reference for users who are constructing searches. It provides details about the search command, including examples of usage, and suggests other commands for you to use.

  1. Navigate to the Search app.
  2. To open Search Assistant, click the green arrow under the search bar.
  3. screenshow_uploadLocal
  4. If the search bar is empty, you'll see a brief description of searching in Splunk and how to construct searches. By default, the assistant displays information for the search command.
  5. The left side of Search Assistant shows a history of the command's usage and what commands were most often used next.
  6. The right side of Search Assistant shows a brief description of the search command and examples of usage.

Add knowledge

Splunk takes search where it's never been before by automatically extracting knowledge from your IT data and letting you add your own knowledge on-the-fly. Add knowledge about the events, fields, transactions, patterns and statistics in your data. You can identify, name and tag this data as well.

Splunk maps all this knowledge at search time, so you can add new fields and event types anytime you need them, without re-indexing the data. Go from finding all events with a particular username, to instantly getting statistics on specific user activities.

Classify similar events

When you search your data, you're essentially weeding out all unwanted events; the results of your search are events that share common characteristics, and you can give them a collective name or "event type". The names of your event types are added as values into an eventtype field. This means that you can search for these groups of events the same way you search for any field. The following example takes you through the steps to save a search as an eventtype and then searching for that field.

If you run frequent searches to investigate SSH and firewall activities, such as sshd logins or firewall denies, you can save these searches as an event type. Also, if you see error messages that are cryptic, you can save it as an event type with a more descriptive name.

  1. Navigate the Search app.
  2. If regularly track SSH activity, such as login attempts, you can save this search as an event type. First, run a search; for example, a search for SSH logins might be:
  3. screenshow_uploadLocal
  4. After you run a search, select the "Event type..." option from the 'Create' dropdown menu. The Save Event Type window appears.
  5. screenshow_uploadLocal
  6. Follow the directions on the screen to name your event type; you might name this event type "sshlogin". Modify the search string if necessary. Optionally, define tags for your event type; this is discussed in more detail later. When you're finished, click "Save".
  7. You can also save event types for other types of SSH activity, such as logouts and timeouts. Now to search for just SSH logins:
  8. screenshow_uploadLocal
  9. If you saved the other saved event types named sshlogout and sshtimeout, you can quickly search for all SSH events:
  10. screenshow_uploadLocal

Extract new fields

Splunk automatically extracts knowledge for you as you index new data; and you can also add new knowledge anytime you need it -- without re-indexing your data. This section shows you how to use the field extractor to interactively extract and save new fields.

  1. Navigate to the Search app.
  2. Run a search for a host, source or sourcetype value. Field extraction for any set of events is linked to the host, source or sourcetype value associated with those events.
  3. Select an event from your results.
  4. Click on the blue dropdown arrow next to the timestamp for this event. This opens a menu.
  5. Select "Extract Fields". This opens the interactive Extract fields window.
  6. screenshow_uploadLocal
  7. Follow the instructions on the page. For example, you might want to add fields for username, source IP, and destination IP.

Tag field values

Tags help you group search results that share field values. A tag is a name that you attach to a particular value of a field such as eventtype, host, source, or sourcetype. For example, you can tag a host's values with a service name or a note indicating compliance with regulations like PCI.

Generally, you can use tags to:

  • Help you track abstract field values, like IP addresses or ID numbers. which you can group with a location or name.
  • Use one tag to group a set of field values together, so you can search on them with one simple command.
  • Give specific extracted fields multiple tags that reflect different aspects of their identity, which enable you to perform tag-based searches that help you quickly narrow down the results you want.

There are two ways to search for a tag - against all fields or a particular fielddialog.

  1. If you want to find any event with a field that was tagged deny:
  2. screenshow_uploadLocal
  3. If you want to find only events with an event type tag, deny:
  4. screenshow_uploadLocal

Learn more about Splunk Knowledge

There's more that you can do to best use and extend Splunk so that it works with your data in a manner that fulfills the needs of your enterprise. You'll want to consult the Knowledge Manager manual as you optimize, maintain, and expand your Splunk deployment over time.

The Knowledge Manager manual teaches you:

  • How to manage and maintain Splunk "knowledge objects" such as events, event types, fields, source types, tags, and transactions.
  • Best practices for working with fields, including lookups and aliasing.
  • Strategies for grouping conceptually related events into transactions.

Monitor and alert

After you use Splunk to identify and locate problems in your system, take advantage of its monitoring and alerting capabilities to keep you notified if those situations recur. Save your searches to run them whenever you want, or set up an alert to do the monitoring for you. Configure alerts to fire when the search results meet conditions that you define. You can even alert on events happening in real time.

Save a search

  1. Create a search that returns results you're interested in.
  2. When your search is done running, select "Save search" from the 'Save' dropdown underneath the search bar:
  3. screenshow_uploadLocal
  4. This pops up the save a search dialog.
  5. screenshow_uploadLocal
  6. Set more options by following the rest of the directions on the page.

Set up an alert

You can turn any search into an alert. Alerts notify you by email or RSS. You can also set up alerts to trigger a script.

  1. If you want to set up your search to be an alert, choose "Alert" from the 'Create' dropdown underneath the search bar:
  2. screenshow_uploadLocal
  3. This pops up the Create Alert dialog:
  4. screenshow_uploadLocal
  5. Follow the directions in the dialog to set up one or more alert methods. Make sure your Splunk server has sendmail (or another MTA) enabled if you want Splunk to email you.

Report and analyze

Create reports with Splunk's built-in visualization tools. Splunk gives you a wide range of options when it comes to reporting. Create simple "top values over time" reports directly from your search results. Use Report Builder to define and format sophisticated charts. Or define reports by hand using Splunk's powerful statistical commands. Finally, you can quickly create dashboards that share your best reports with others.

Create a simple report

After you run a search you can quickly launch reports providing basic information about the fields in your search results.

  1. Navigate to the Search app.
  2. Run a search for any term or field you want to report on.
  3. When your search is done find a field in the search results sidebar that you would like to report on and click it. For example, pick UID if you want to report on a set of user IDs.
  4. screenshow_uploadLocal
  5. A popup window appears, displaying information about the field you've selected. You can launch a report for each field. Select a report you'd like to run, such as "Average over time" or "Top values overall."
  6. screenshow_uploadLocal
  7. Report Builder appears in a separate window, showing a chart based on the event data returned by your search. From here you can reformat the report, save it, print it, and more.

Use Report Builder

Launch the report builder to create and format your reports.

  1. Navigate to the Search app.
  2. Run a search for any term or field you want to report on.
  3. In the 'Create' menu over the timeline, you'll see a link to create a report. Click this link to launch the Report Builder. (You don't need to wait for the search to complete before launching the Report Builder.)
  4. screenshow_uploadLocal
  5. You can also create a report from directly within the timeline view by clicking on the Results Chart button at the top of the Results area.
  6. screenshow_uploadLocal

Use reporting commands

When you use the Report Builder drop-down lists to define a report, you may notice that Splunk updates the Report Builder search box with the statistical reporting commands Splunk uses to run the report. This section explains how to use these reporting commands directly from the search bar.

  1. Navigate to the Search app.
  2. Run a search for any term in your data.
  3. Follow your search terms with a "pipe" character and some basic reporting commands. This basic report, for example, finds the top 5 most common sources in your internal Splunk index:
  4. screenshow_uploadLocal
  5. Select the Results Chart button underneath the timeline to see your results as a chart. Reformat the report if you wish using the 'Formatting options.'
  6. If you want to build a more sophisticated report, there's more information about Splunk reporting commands in Splunk's User Manual.

Preview reports

When you run a report, Splunk can preview the report results for you as the search runs. This feature saves you time, especially when running searches across large time periods. Note that report preview is enabled by default for searches that use reporting commands, so try it out on a report over a large period of time.

  1. Navigate to the Search app.
  2. Enter a report-generating search into the search bar.
  3. While the report is running, you'll see a preview show up in the results area.
  4. If you're not happy with your report and want to change it before it finishes running, just click the cancel button and edit the search string.

Build dashboards

Save and share your searches and reports by creating a dashboard. Dashboards are a place to collect your most useful and informative reports, and make them available to other users. Start a dashboard from scratch, or create one as you create reports.

  1. Navigate to the Search app.
  2. If you’ve already run a search or report, under the Create menu, clicking "Dashboard panel…" will allow you to add your search or report to an existing dashboard, or create a new dashboard.
  3. screenshow_uploadLocal

You can also start by creating your dashboard from scratch.

  1. From the Dashboards & Views menu, choose "Create dashboard":
  2. screenshow_uploadLocal
  3. The Create new dashboard window appears. Give the dashboard a short name (Dashboard ID) and long name (Dashboard name). Click Create when you're done.
  4. At first, your dashboard is empty. Click Edit the dashboard to open the Edit window, and choose the panel type and saved search for your first panel. Click Add panel to add your new panel to the dashboard.
  5. Click Edit panel if you want to rename the panel, change its format, or update the search it's based on.
  6. Repeat the last two steps to create more panels. Drag them around until you have them set up the way you want them.
  7. You're finished! Click Close to see how your dashboard looks. If you see stuff you want to change, go to the Action menu at the top of the page, select Edit dashboard..., and fix them.

More

Looking for more information on what you can do with Splunk? Here are a few more links to Splunk's online documentation.