* Please Note: As of Mikrotik RouterOS 7.x. IP Accounting functionality has been removed and is unavailable as a function on Datatill.
IP Accounting Guide
1. Overview
DataTill normally uses radius accounting to track the data usage for each individual user. This means that there needs to be a radius account in DataTill as well as a PPPoE authenticated radius session on a router on the network. DataTill then reads the FreeRADIUS generated accounting info to process usage, and auto blocks the PPPoE account from authentication once it is capped, so uses radius speed attributes to throttle the user once a soft cap limit has been reached.
When IP Accounting is used, there is no PPPoE dialup and no FreeRADIUS session involved. This means that the usage information is pulled from the High site router the user is connected to. This is done by using the MikroTik IP Accounting feature, which maintains a table in memory for each source and destination IP Pair.
Speed limits are enforced by creating static queues for each radius user’s fixed IP on a specified breakout router. Once a user has reached a soft or hard cap limit, the static queue needs to be adjusted accordingly. For this to work each DataTill radius user account that uses IP Accounting must have a fixed IP address assigned. To ensure no conflicting IP addresses are used, IP address lists will be maintained and assigned to individual High site routers. User accounts will then be able to select available IPs from these lists.
Capped and suspended accounts will be blocked by a firewall rule on the breakout routers, which will redirect traffic from all users in special capped or suspended address lists. This means that it is crucial that all router firewalls are set up correctly.
2. Enabling the IP Accounting module in DataTill
The IP Accounting module can be turned on or off on a global level. If the module is disabled, there will be no IP Accounting switches or settings visible on the system.
To enable the IP accounting module in DataTill, go to “Devices” > “Device Monitoring Setup” and then click on the “IP Accounting” option.
You will see the following on the screen once it has finished loading:
Click on the “Enable. MikroTik IP Accounting” button to enable the module. After the MikroTik
IP Accounting function has been enabled; more settings will appear.
When using IP Accounting please note the following:
Datatill will enable IP accounting on each Mikrotik router (via the mikrotik API) where accounting data is being fetched from. Consider fetching IP Accounting from the highsite router where the user connects, rather than from the breakout router. Queues to limit the user speed will be auto created on the router where the user’s IP address was last detected. User (radius) accounts that needs to use IP accounting instead or radius to count usage needs to be switched from radius to IP accounting. All data accounts using IP accounting must have a static IP address assigned in order to link accounting data to data accounts.
2.1. IP Accounting Settings:
2.1.1. Default IP Pair Threshold
Here you can add the IP Pair Threshold. If the router is old or has outdated firmware, the threshold will be lower. New routers tend to be able to take a higher threshold. It is advised to always allow the maximum number of IP Pairings. Please see section 3 for more information.
2.1.2. Fetch Method
There are three distinct methods available. To fetch IP accounting information from the routers:
- MikroTik API
- Router Accounting URL
- Router Scheduler Script and FTP
The “Fetch Method” chosen is used as the default for all routers but can be overridden on individual routers is required. For more information on which fetch method to choose, please refer to section 4 (Monitoring and collecting IP Accounting usage information).
2.1.3. Default Web URL Port:
This section is only applicable should you choose to use the Router Accounting URL as your preferred fetch method. The default port will always be on 80. In the case that you have made any manual changes to the router, please remember to adjust the port in the IP Accounting set up to the correct port.
Once the default port has been set, it will automatically pull through to all new devices that are installed onto the network. Please note that this is only the default port number and that individual routers on the network can have different port numbers.
2.1.4. Default Scheduler File Creation Interval
This section is only applicable should you choose to make use of the Router scheduler script & FTP setting fetch method. This number equals the number of seconds you would like to extract data for. Ideally, this number should not be higher than 60 (1 minute) as the system’s cron job interval is 60 seconds.
In short, the shorter the time is set for, the more files the script will extract from the router which will lead to more processing and strain on your server. However, you should know that the less files are extracted over a longer period of time (depending on how busy the line is versus the amount of time set), the greater your chances are of losing traffic data.
2.1.5. User Data for Traffic Identification
When enabled (together with extended usage logging) all traffic will be logged for classification, including radius, hotspot, etc., not just IP Accounting based accounts. The traffic will be stored in the stats database and the breakdown will be available in each customer’s usage portal as well as the system usage report.
Example:
Apple, Browsing, Facebook, Google, Mail, Video, Other and Unclassified.
The intensity of the analysis can be configured under the extended logging system settings. Please note that this will however add significant processing overhead to the server.
2.1.6. Fetch IP Accounting Data from All MikroTik Routers
When enabled all Mikrotik devices defined under Network Devices will be scanned for IP Accounting data. The frequency of the scans can be configured via the relevant cron job When disabled you need to enable IP accounting on each individual Network Device and configure the device accordingly.
Please note that if a highsite router is assigned to a specific user who has the IP Accounting enabled, that the IP Accounting will then automatically be enabled for the High site router.
2.1.6.1. Enable All MikroTik Routers
If you would like to enable the “Fetch IP Accounting Data from all MikroTik Routers” toggle switch. Once enabled, a scheduled job will try to pull accounting information from all MikroTik routers defined in the system.
2.1.6.2. Enable individual MikroTik Routers
For more information on how to enable and disable individual MikroTik routers, please refer to 3.1 and 3.2.
2.1.7. NTP Timer Server IP:
All files that are exported from the router will be saved onto a local server and the file name will be saved with a certain date and time. This means that the date and time on the router should be set up correct to ensure that the files are not saved incorrectly. To do this, you will need to have a time server set up. It is preferred that the time server should be within your network. After your time server is set up, add the IP address to the NTP Time Server IP field and click on the update button. Please note that this is a mandatory field for all ISP’s using the IP Accounting module.
Should you choose to only enable individual routers, you will need to update each router by clicking on the “Update Router” button to ensure that all the settings have synced to the router. When clicking on the “Update Router” button, the NTP time setting will automatically be added to that individual router.
2.2. Capped Page setup
All capped packages that run through the IP Accounting module, will be managed by a router or different routers (depending on your infrastructure). These routers will usually be your breakout routers.
For the IP Accounting module to successfully cap packages, you will need to configure the firewall on your router(s). If the firewall on your routers is not configured, you will not be able to cap the customers and they will be able to continue to surf the internet.
We suggest that you set up your routers.
firewalls in one of the following ways:
- Either block the customer off the internet.
- Redirect the customer to a hotspot which will enable them to top-up.
- Redirect the customer to a proxy with a static page.
| Green: | In the green section, you can choose whether you want to enable the capped IP address lists. |
| Red: | In this section, you will add a name for your capped lists. DataTill will automatically add all capped IP Addresses as well as automatically remove all IP addresses that has been topped up. |
| Yellow: | Here you will select the routers on which your capped lists should be maintained. This will usually be your breakout routers. Example: NAS.FF.0002-101.10.0.199 |
2.3. Blocked Page Setup
If enabled the address list defined below will be maintained on the routers below with the IP addresses of capped data accounts.
Please Note: If you manually add an IP to the address lists and add a comment then these entries WILL NOT be removed from the address list when the data accounts are uncapped. Only IP Accounting and radius based capped accounts with fixed IP addresses and data products that are not uncapped and are not set to auto rollover will be included.
2.3.1. Blocked Adress List Name
This will be the name of the blocked address list on each router.
2.3.2. Devices
This will be the devices (Mikrotik Routers) on which to maintain the blocked address list.
2.4. Dynamic IP Queue Setup
Just like with your capped page setup, your queue setup will also take place through your breakout routers. If the routers are not set up correctly, the customer’s speed limits will not be enforced. When configuring the router, you can set up the capped and speed queues on the same router or on different routers.
If enabled individual queues will be maintained for each user IP on the routers below. If disable, then no speed limits will be enforced for IP Accounting based user accounts.
| Blue: | In this section, you can choose whether you want to enable or disable the queue. |
| Purple: | In this section, you will add a name to your capped lists. DataTill will automatically add all capped IP Addresses as well as automatically remove all IP addresses that has been topped up. |
Please note that DataTill has a background job that resets all queues to their original state every 5 minutes. After your changes have been made, please click on the “Update” button to save your changes.
3. MikroTik router configuration management:
DataTill will automatically periodically check all routers set to use. IP accounting and enable the settings on the router if it is not enabled. If the schedule script method is enabled, then DataTill will also verify that the script and correct schedule are defined on the various routers.
To find a list of all network devices, go to “Devices” > “Network Devices” and then click on the “Network Devices” option.
After the screen is finished loading, look for the router that you would like to enable to IP Accounting functionality for. Once you have found the router, click on the blue edit button found on the right-hand side of the screen Navigate to IP Pools section and you can edit or add your IP Pool.
3.1. Enabling routers individually
After clicking on the edit button found on the right-hand side of the device, you will the following pop-up screen. Click on the “IP Accounting” tab to start editing the specific router properties.
| Red: | In this section, you can enable the IP Accounting functionality for the specific device. |
| Yellow: | Here you can add the IP Pair Threshold. If the router is old or has outdated firmware, the threshold will be lower. New routers tend to be able to take a higher threshold. It is advised to always allow the maximum number of IP Pairings. If the router is unable to handle such a large amount of IP Pairings, you will receive an error notification when clicking on the black “Update Router” button. |
| Dark blue: | In this section, you can choose the fetch method. |
| Purple: | This is where you need to add the scheduler file creation interval. This can be anything from 10 – 60 seconds. |
| Black: | After filling in all the details, it is crucial that you first click on the “Update Router” button before. |
| Green: | In this section, you will be able to view all IP accounting files that have been extracted from this specific router. |
After all details have been filled in and you have clicked on the update router button, click on the “Save Changes” button.
3.2. Disabling routers individually
After clicking on the edit button found on the right-hand side of the device, you will the following pop-up screen. Click on the “IP Accounting” tab to start editing the specific router properties.
| Red: | To disable the router for IP Account tracking, ensure that the toggle switch is on “Off”. |
| Purple: | Click on the “Update Router” button to ensure changes are saved to the router. |
| Green: | After your changes has been made and you have clicked on the update router button, click on the “Save Changes” button. |
3.3. Maintaining user speed limits
It is important that all queues and caps are properly setup on the breakout router as the customer’s queues and cap will be managed through the customer’s IP address. Even though helpdesk agents can adjust the customer’s queues, DataTill runs a background job that resets all queues to their original state every 5 minutes.
DataTill will only allow the following factors to influence the customer’s queue:
- Package Changes
- Capped Packages
- Top-ups
- Uncapped Packages
- Soft-capped Packages
- Adding New Accounts
- Expiring Accounts
3.3.1. Capped Accounts
DataTill will ensure that an IP based queue is created on these routers for every IP Accounting based user account. Once a user account’s speed is adjusted in DataTill, is capped, or reaches a soft limit, their queue will be adjusted accordingly on these routers. As the ‘queue maintenance’ is run as a background job it may take up to two minutes before speed limits are adjusted in line with the package changes made to the user account.
3.3.2. Sub Accounts
When sub-user accounts are created in DataTill (for example a home and business account share the same package) then both accounts need to share the same data and speed limits. In these scenarios, a queue with multiple IP addresses will be created. This multi-IP queue will include the IP address of the parent account, as well as the IP address of each child account. MikroTik will then enforce the speed limit across these accounts, assuming all the accounts use the same breakout router.
3.3.3. Hard capped and suspended accounts
Accounts that need to be blocked (hard capped, suspended and expired) will use a firewall rule on the breakout router to redirect the user to a web proxy page displaying a static age. To achieve this DataTill will automatically add or remove capped and blocked user IP addresses from specific address lists used by the firewall rules, on the breakout routers.
4. Monitoring and collecting IP Accounting usage information
To read IP Accounting usage information, the MikroTik router where the user connects, needs to have IP Accounting enabled. Once enabled, the router will keep a memory table of IP pairs, containing the source and destination IPs, as well as the data consumed. After this data is read, the table will be cleared. This table has a limit of only 8192 pairs.
Note: that some routers allow a larger value of up to 262144 entries. Once the limit is reached the additional IP pair information is discarded, so it is important to read this information frequently enough so that the limit is not exceeded, and no information is lost.
4.1. How does DataTill know which queues to check?
On the router, you will be able to find a list of all queues. All the queues, for each IP accounting user, will have a unique identifier in the form of a pre-fix. DataTill will only look at the queues that start with DT_AUTO_QUE_USERNAME OF CUSTOMER ACCOUNT.
Please note that you can only have one queue per IP address. If you have more than one queue per IP address, then DataTill will eliminate the IP address and que that does not have the unique identifier prefix. When setting up the user account and IP, you will need to ensure that you only choose IP’s that are allocated to the IP Accounting pool. All user accounts with IP’s outside of this pool will also be eliminated.
4.2. Choosing a fetch method:
As previously mentioned, there are three methods in which DataTill can retrieve the IP Accounting information from the relevant MikroTik routers:
- MikroTik API
- Router Accounting URL
- Router Scheduler Script and FTP
4.2.1. MikroTik API
DataTill will log into the router via the MikroTik API and then retrieve the IP Accounting table data stored on the router.
4.2.2. Router Accounting URL
DataTill calls a local URL on the router, which will return a csv-based contents of the IP Accounting table data. DataTill will automatically set the IP Accounting URL.
4.2.3. Router Scheduler Script and FTP (Scheduled file dumping)
A local script is created on the MikroTik router that dumps IP Accounting data to a local file every few seconds (30-60). The script checks available free disk space to ensure that it will not dump data if there is less than a predefined amount (2MB) of disk space available.
DataTill then uses FTP (File Transfer Protocol) to retrieve the files from the router and deletes the files once they are downloaded to the server. The last file on the router will always be skipped in case the file is still being written to at the time of download.
4.2.4. Suggested method
All the above methods can be used, either exclusively or in combination. It is however preferred that the scheduled file dumping option is chosen as it is the last likely to result in lost usage data due to the memory table limit being reached. To facilitate the accurate configuration of the script and scheduler on the routers, DataTill will auto configure the relevant routers when this method is chosen.
Once the usage data has been collected, it is processed exactly like radius-based usage information. One of the IP addresses from the IP Pair is matched to a radius account using the fixed IP address assigned to each of the IP Accounting based user accounts. Data is then summarized on a daily and monthly basis and made available in the end user’s usage portal just like Radius, Fibre and LTE based usage data. If extended logging is enabled within DataTill then the remote IP addresses are used to identify and classify the data usage (Dropbox, Facebook, YouTube, etc.), like the way net flow data is analyzed. This allows the customer to view a breakdown of data usage in their end user portal. Please note that extended logging may be very resource intensive on your DataTill server.
5. Managing user IP address assignments
Every IP Accounting based user account must use a fixed IP address. Two users cannot share the same IP address, as the system will then be unable to allocate data usage to the correct account.
5.1. Adding IP Pools to a router
To add an IP Pool to a router, edit the router properties of the specific router that you would like to work on. Once the pop-up screen has loaded, go to the “IP Pools” tab.
| Blue: | To add an IP Pool to the router, click on the blue “Add” button. |
| Purple: | In this section, you will be able to view a list of all IP Pools linked to this router. |
| Green: | After making any changes, remember to click on the “Save Changes” button. |
5.2. High site IP ranges
One or more IP subnets will be assigned to each High site router from within DataTill. The system will automatically count the number of available IP addresses, and the admin user will be able to exclude reserved IPs from the list of selectable IP addresses, like for example: equipment on the High site such as cameras and power monitors.
5.3. User IP Allocation
When IP Accounting user accounts are created, the High site where the user will connect must be selected. Once the High site has been chosen, the list of available subnets will be available for selection. Once a subnet has been chosen, a free IP can be selected from the list within that subnet.
5.4. IP Exclusions
DataTill will automatically keep track of which IP’s are already in use amongst all the IP Accounting user accounts and will prevent you from selecting an IP that is in use (already allocated) or excluded from selection. IP addresses assigned to other network devices (routers, radios, and power monitors) will also be excluded form selection. DataTill will also indicate whether the IP address selected is live on the network, by doing a ping test.
The method of managing these IP ranges will be like the way Radius IP pools are being managed.
6. Creating IP Accounting user accounts
IP Accounting based user accounts are created the same as radius accounts. On the create user pop-up window there is a selector to choose between radius or IP accounting-based data traffic counting. (Note that this option is only visible if IP accounting has been enabled in the system)
To create the IP Accounting user account, go to the customer profile (“edit customer” screen) and then “add” a new user account in the “Data Accounts” section.
After clicking on the “add” button, you will see the following screen:
| Red: | In this section, you will add the customer’s username. After adding a username, click on the generate password button. |
| Yellow: | In this section, you will need to choose whether you want the traffic to be counted via radius accounting or MikroTik IP Accounting. When adding a normal radius account, you will leave the setting on Radius Accounting. When adding an IP Accounting user account, you will need to change the setting to MikroTik IP Accounting. |
| Dark blue: | Here you will choose the data product as per the customer request. |
| Light Blue: | Here you will be able to insert a fixed IP address from the IP Pool that has been allocated to the specific high site. |
| Black: | Always ensure that the authentication is accepted. |
| Green: | When all details have been filled in, click on the “Add Radius User” button. |
Data package selection, pricing and top-up settings are done in the same way as which you would have added these settings for a normal radius account. Please note that when adding sub-accounts, they need to follow the same traffic counting method as the parent account.
7. IP Accounting based billing
7.1. Monthly billing
Billing for IP Accounting based user accounts is identical to other data accounts like radius, LTE and Openserve. Monthly recurring billing is generated from the data package cost and can be overridden on an individual basis.
7.2. Suspensions
When IP accounting-based users are suspended, the user’s IP address is added to a blocked account address list, on a designated router. A firewall rule on that router needs to be configured so that any traffic from any IP in that address list is redirected to a proxy page indicating a suspended accounts message.
If multiple breakout routers are in use, then the address lists will need to be auto maintained on all these routers.
7.3. Capped accounts
Similar to suspended accounts, hard capped account IP addresses are also added to a central address list on the breakout router, where a similar firewall rule needs to redirect the user to a similar page showing that the user’s cap has been reached.