mirrors
/
AdGuardHome
mirror of https://github.com/AdguardTeam/AdGuardHome.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Pull request 1736: 4299-querylog-stats-api 

Merge in DNS/adguard-home from 4299-querylog-stats-api to master

Updates #1717.
Updates #4299.

Squashed commit of the following:

commit 5b706b7997
Merge: 48b62b0f 306c1983
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 22 13:53:09 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit 48b62b0f18
Author: Vladislav Abdulmyanov <v.abdulmyanov@adguard.com>
Date:   Wed Mar 22 12:25:04 2023 +0200

    client: fix styles, add titles and descrs

commit 97e31cff70
Author: Vladislav Abdulmyanov <v.abdulmyanov@adguard.com>
Date:   Tue Mar 21 18:38:12 2023 +0200

    client: add ignored domains for querylog

commit 24d75c4376
Author: Vladislav Abdulmyanov <v.abdulmyanov@adguard.com>
Date:   Tue Mar 21 18:21:13 2023 +0200

    client: add ignore domains for stats

commit eefc3891d0
Merge: 978675ea 1daabb97
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Tue Mar 21 10:53:35 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit 978675ea2c
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Tue Mar 21 10:53:11 2023 +0300

    openapi: fix chlog

commit 2ed33007aa
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Mar 20 17:49:07 2023 +0300

    home: fix tests

commit 6af11520c1
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Mar 20 17:40:16 2023 +0300

    home: fix typo

commit 56acdfde5b
Merge: 319da34d 48431f8b
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Mar 20 17:32:58 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit 319da34de4
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Fri Mar 3 17:34:38 2023 +0300

    querylog: fix docs

commit d5a8f24d5b
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Fri Mar 3 11:42:00 2023 +0300

    stats: fix docs

commit e0cbfc1c40
Merge: 4743c810 012e5beb
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 1 18:45:17 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit 4743c81038
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 1 18:14:16 2023 +0300

    all: imp code; fix time conversion

commit 34310cffd7
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 1 12:34:11 2023 +0300

    chlog: restore order

commit cadd864a66
Merge: 2f3e25be bb226434
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 1 12:26:06 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit 2f3e25bee5
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Wed Mar 1 12:25:14 2023 +0300

    all: fix fmt

commit d54022baa6
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Tue Feb 28 16:16:40 2023 +0300

    all: imp code; fix chlog

commit df22de91f5
Merge: e1ea4797 a772212d
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Feb 27 17:24:09 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit e1ea4797af
Merge: d7db0a5a bb80a7c2
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Feb 27 17:23:20 2023 +0300

    Merge branch 'master' into 4299-querylog-stats-api

commit d7db0a5af1
Author: Stanislav Chzhen <s.chzhen@adguard.com>
Date:   Mon Feb 27 17:12:20 2023 +0300

    all: imp docs

... and 15 more commits
This commit is contained in:
Stanislav Chzhen 2023-03-23 13:46:57 +03:00
parent 306c1983a2
commit 143616ca6e
29 changed files with 995 additions and 157 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
CHANGELOG.md
View File

@ -25,6 +25,12 @@ NOTE: Add new changes BELOW THIS COMMENT.
### Added
- Two new HTTP APIs, `PUT /control/stats/config/update` and `GET
control/stats/config`, which can be used to set and receive the query log
configuration. See openapi/openapi.yaml for the full description.
- Two new HTTP APIs, `PUT /control/querylog/config/update` and `GET
control/querylog/config`, which can be used to set and receive the statistics
configuration. See openapi/openapi.yaml for the full description.
- The ability to set custom IP for EDNS Client Subnet by using the DNS-server
configuration section on the DNS settings page in the UI ([#1472]).
- The ability to manage safesearch for each service by using the new
@ -37,8 +43,26 @@ NOTE: Add new changes BELOW THIS COMMENT.
#### Configuration Changes
In this release, the schema version has changed from 17 to 19.
In this release, the schema version has changed from 17 to 20.
- Property `statistics.interval`, which in schema versions 19 and earlier used
to be an integer number of days, is now a string with a human-readable
duration:
```yaml
# BEFORE:
'statistics':
# …
'interval': 1
# AFTER:
'statistics':
# …
'interval': '24h'
```
To rollback this change, convert the property back into days and change the
`schema_version` back to `19`.
- The `dns.safesearch_enabled` field has been replaced with `safe_search`
object containing per-service settings.
- The `clients.persistent.safesearch_enabled` field has been replaced with
@ -64,6 +88,23 @@ In this release, the schema version has changed from 17 to 19.
client's specific `clients.persistent.safesearch` and then change the
`schema_version` back to `17`.
### Deprecated
- The `GET /control/stats_info` HTTP API; use the new `GET
/control/stats/config` API instead.
**NOTE:** If interval is custom then it will be equal to `90` days for
compatibility reasons. See openapi/openapi.yaml and `openapi/CHANGELOG.md`.
- The `POST /control/stats_config` HTTP API; use the new `PUT
/control/stats/config/update` API instead.
- The `GET /control/querylog_info` HTTP API; use the new `GET
/control/querylog/config` API instead.
**NOTE:** If interval is custom then it will be equal to `90` days for
compatibility reasons. See openapi/openapi.yaml and `openapi/CHANGELOG.md`.
- The `POST /control/querylog_config` HTTP API; use the new `PUT
/control/querylog/config/update` API instead.
### Fixed
- Panic caused by empty top-level domain name label in `/etc/hosts` files
@ -103,8 +144,6 @@ See also the [v0.107.26 GitHub milestone][ms-v0.107.26].
#### Configuration Changes
In this release, the schema version has changed from 16 to 17.
- Property `edns_client_subnet`, which in schema versions 16 and earlier used
to be a part of the `dns` object, is now part of the `dns.edns_client_subnet`
object:

4
client/src/__locales/en.json
View File

@ -525,6 +525,10 @@
"statistics_retention_confirm": "Are you sure you want to change statistics retention? If you decrease the interval value, some data will be lost",
"statistics_cleared": "Statistics successfully cleared",
"statistics_enable": "Enable statistics",
"ignore_domains": "Ignored domains (separated by newline)",
"ignore_domains_title": "Ignored domains",
"ignore_domains_desc_stats": "Queries for these domains are not written to the statistics",
"ignore_domains_desc_query": "Queries for these domains are not written to the query log",
"interval_hours": "{{count}} hour",
"interval_hours_plural": "{{count}} hours",
"filters_configuration": "Filters configuration",

2
client/src/actions/queryLogs.js
View File

@ -177,7 +177,7 @@ export const getLogsConfigSuccess = createAction('GET_LOGS_CONFIG_SUCCESS');
export const getLogsConfig = () => async (dispatch) => {
dispatch(getLogsConfigRequest());
try {
const data = await apiClient.getQueryLogInfo();
const data = await apiClient.getQueryLogConfig();
dispatch(getLogsConfigSuccess(data));
} catch (error) {
dispatch(addErrorToast({ error }));

2
client/src/actions/stats.js
View File

@ -13,7 +13,7 @@ export const getStatsConfigSuccess = createAction('GET_STATS_CONFIG_SUCCESS');
export const getStatsConfig = () => async (dispatch) => {
dispatch(getStatsConfigRequest());
try {
const data = await apiClient.getStatsInfo();
const data = await apiClient.getStatsConfig();
dispatch(getStatsConfigSuccess(data));
} catch (error) {
dispatch(addErrorToast({ error }));

20
client/src/api/Api.js
View File

@ -497,9 +497,9 @@ class Api {
// Settings for statistics
GET_STATS = { path: 'stats', method: 'GET' };
STATS_INFO = { path: 'stats_info', method: 'GET' };
GET_STATS_CONFIG = { path: 'stats/config', method: 'GET' };
STATS_CONFIG = { path: 'stats_config', method: 'POST' };
UPDATE_STATS_CONFIG = { path: 'stats/config/update', method: 'PUT' };
STATS_RESET = { path: 'stats_reset', method: 'POST' };
@ -508,13 +508,13 @@ class Api {
return this.makeRequest(path, method);
}
getStatsInfo() {
const { path, method } = this.STATS_INFO;
getStatsConfig() {
const { path, method } = this.GET_STATS_CONFIG;
return this.makeRequest(path, method);
}
setStatsConfig(data) {
const { path, method } = this.STATS_CONFIG;
const { path, method } = this.UPDATE_STATS_CONFIG;
const config = {
data,
};
@ -529,9 +529,9 @@ class Api {
// Query log
GET_QUERY_LOG = { path: 'querylog', method: 'GET' };
QUERY_LOG_CONFIG = { path: 'querylog_config', method: 'POST' };
UPDATE_QUERY_LOG_CONFIG = { path: 'querylog/config/update', method: 'PUT' };
QUERY_LOG_INFO = { path: 'querylog_info', method: 'GET' };
GET_QUERY_LOG_CONFIG = { path: 'querylog/config', method: 'GET' };
QUERY_LOG_CLEAR = { path: 'querylog_clear', method: 'POST' };
@ -543,13 +543,13 @@ class Api {
return this.makeRequest(url, method);
}
getQueryLogInfo() {
const { path, method } = this.QUERY_LOG_INFO;
getQueryLogConfig() {
const { path, method } = this.GET_QUERY_LOG_CONFIG;
return this.makeRequest(path, method);
}
setQueryLogConfig(data) {
const { path, method } = this.QUERY_LOG_CONFIG;
const { path, method } = this.UPDATE_QUERY_LOG_CONFIG;
const config = {
data,
};

36
client/src/components/Settings/LogsConfig/Form.js
View File

@ -4,18 +4,28 @@ import { Field, reduxForm } from 'redux-form';
import { Trans, withTranslation } from 'react-i18next';
import flow from 'lodash/flow';
import { CheckboxField, renderRadioField, toFloatNumber } from '../../../helpers/form';
import { FORM_NAME, QUERY_LOG_INTERVALS_DAYS } from '../../../helpers/constants';
import {
CheckboxField,
renderRadioField,
toFloatNumber,
renderTextareaField,
} from '../../../helpers/form';
import {
FORM_NAME,
QUERY_LOG_INTERVALS_DAYS,
HOUR,
DAY,
} from '../../../helpers/constants';
import '../FormButton.css';
const getIntervalTitle = (interval, t) => {
switch (interval) {
case 0.25:
case 6 * HOUR:
return t('interval_6_hour');
case 1:
case DAY:
return t('interval_24_hour');
default:
return t('interval_days', { count: interval });
return t('interval_days', { count: interval / DAY });
}
};
@ -66,6 +76,22 @@ const Form = (props) => {
{getIntervalFields(processing, t, toFloatNumber)}
</div>
</div>
<label className="form__label form__label--with-desc">
<Trans>ignore_domains_title</Trans>
</label>
<div className="form__desc form__desc--top">
<Trans>ignore_domains_desc_query</Trans>
</div>
<div className="form__group form__group--settings">
<Field
name="ignored"
type="textarea"
className="form-control form-control--textarea font-monospace text-input"
component={renderTextareaField}
placeholder={t('ignore_domains')}
disabled={processing}
/>
</div>
<div className="mt-5">
<button
type="submit"

10
client/src/components/Settings/LogsConfig/index.js
View File

@ -10,13 +10,15 @@ class LogsConfig extends Component {
const { t, interval: prevInterval } = this.props;
const { interval } = values;
const data = { ...values, ignored: values.ignored ? values.ignored.split('\n') : [] };
if (interval !== prevInterval) {
// eslint-disable-next-line no-alert
if (window.confirm(t('query_log_retention_confirm'))) {
this.props.setLogsConfig(values);
this.props.setLogsConfig(data);
}
} else {
this.props.setLogsConfig(values);
this.props.setLogsConfig(data);
}
};
@ -30,7 +32,7 @@ class LogsConfig extends Component {
render() {
const {
t, enabled, interval, processing, processingClear, anonymize_client_ip,
t, enabled, interval, processing, processingClear, anonymize_client_ip, ignored,
} = this.props;
return (
@ -45,6 +47,7 @@ class LogsConfig extends Component {
enabled,
interval,
anonymize_client_ip,
ignored: ignored.join('\n'),
}}
onSubmit={this.handleFormSubmit}
processing={processing}
@ -62,6 +65,7 @@ LogsConfig.propTypes = {
enabled: PropTypes.bool.isRequired,
anonymize_client_ip: PropTypes.bool.isRequired,
processing: PropTypes.bool.isRequired,
ignored: PropTypes.array.isRequired,
processingClear: PropTypes.bool.isRequired,
setLogsConfig: PropTypes.func.isRequired,
clearLogs: PropTypes.func.isRequired,

48
client/src/components/Settings/StatsConfig/Form.js
View File

@ -4,23 +4,31 @@ import { Field, reduxForm } from 'redux-form';
import { Trans, withTranslation } from 'react-i18next';
import flow from 'lodash/flow';
import { renderRadioField, toNumber, CheckboxField } from '../../../helpers/form';
import { FORM_NAME, STATS_INTERVALS_DAYS, DISABLED_STATS_INTERVAL } from '../../../helpers/constants';
import {
renderRadioField,
toNumber,
CheckboxField,
renderTextareaField,
} from '../../../helpers/form';
import {
FORM_NAME,
STATS_INTERVALS_DAYS,
DAY,
} from '../../../helpers/constants';
import '../FormButton.css';
const getIntervalTitle = (interval, t) => {
switch (interval) {
const getIntervalTitle = (intervalMs, t) => {
switch (intervalMs / DAY) {
case 1:
return t('interval_24_hour');
default:
return t('interval_days', { count: interval });
return t('interval_days', { count: intervalMs / DAY });
}
};
const Form = (props) => {
const {
handleSubmit,
change,
processing,
submitting,
invalid,
@ -38,13 +46,6 @@ const Form = (props) => {
component={CheckboxField}
placeholder={t('statistics_enable')}
disabled={processing}
onChange={(event) => {
if (event.target.checked) {
change('interval', STATS_INTERVALS_DAYS[0]);
} else {
change('interval', DISABLED_STATS_INTERVAL);
}
}}
/>
</div>
<label className="form__label form__label--with-desc">
@ -65,15 +66,26 @@ const Form = (props) => {
placeholder={getIntervalTitle(interval, t)}
normalize={toNumber}
disabled={processing}
onChange={(event) => {
if (event.target.checked) {
change('enabled', true);
}
}}
/>
))}
</div>
</div>
<label className="form__label form__label--with-desc">
<Trans>ignore_domains_title</Trans>
</label>
<div className="form__desc form__desc--top">
<Trans>ignore_domains_desc_stats</Trans>
</div>
<div className="form__group form__group--settings">
<Field
name="ignored"
type="textarea"
className="form-control form-control--textarea font-monospace text-input"
component={renderTextareaField}
placeholder={t('ignore_domains')}
disabled={processing}
/>
</div>
<div className="mt-5">
<button
type="submit"

15
client/src/components/Settings/StatsConfig/index.js
View File

@ -6,9 +6,13 @@ import Card from '../../ui/Card';
import Form from './Form';
class StatsConfig extends Component {
handleFormSubmit = (values) => {
handleFormSubmit = ({ enabled, interval, ignored }) => {
const { t, interval: prevInterval } = this.props;
const config = { interval: values.interval };
const config = {
enabled,
interval,
ignored: ignored ? ignored.split('\n') : [],
};
if (config.interval < prevInterval) {
if (window.confirm(t('statistics_retention_confirm'))) {
@ -29,7 +33,7 @@ class StatsConfig extends Component {
render() {
const {
t, interval, processing, processingReset,
t, interval, processing, processingReset, ignored, enabled,
} = this.props;
return (
@ -42,7 +46,8 @@ class StatsConfig extends Component {
<Form
initialValues={{
interval,
enabled: !!interval,
enabled,
ignored: ignored.join('\n'),
}}
onSubmit={this.handleFormSubmit}
processing={processing}
@ -57,6 +62,8 @@ class StatsConfig extends Component {
StatsConfig.propTypes = {
interval: PropTypes.number.isRequired,
ignored: PropTypes.array.isRequired,
enabled: PropTypes.bool.isRequired,
processing: PropTypes.bool.isRequired,
processingReset: PropTypes.bool.isRequired,
setStatsConfig: PropTypes.func.isRequired,

6
client/src/components/Settings/index.js
View File

@ -98,6 +98,7 @@ class Settings extends Component {
<div className="col-md-12">
<LogsConfig
enabled={queryLogs.enabled}
ignored={queryLogs.ignored}
interval={queryLogs.interval}
anonymize_client_ip={queryLogs.anonymize_client_ip}
processing={queryLogs.processingSetConfig}
@ -109,6 +110,8 @@ class Settings extends Component {
<div className="col-md-12">
<StatsConfig
interval={stats.interval}
ignored={stats.ignored}
enabled={stats.enabled}
processing={stats.processingSetConfig}
processingReset={stats.processingReset}
setStatsConfig={setStatsConfig}
@ -139,6 +142,8 @@ Settings.propTypes = {
stats: PropTypes.shape({
processingGetConfig: PropTypes.bool,
interval: PropTypes.number,
enabled: PropTypes.bool,
ignored: PropTypes.array,
processingSetConfig: PropTypes.bool,
processingReset: PropTypes.bool,
}),
@ -149,6 +154,7 @@ Settings.propTypes = {
processingSetConfig: PropTypes.bool,
processingClear: PropTypes.bool,
processingGetConfig: PropTypes.bool,
ignored: PropTypes.array,
}),
filtering: PropTypes.shape({
interval: PropTypes.number,

9
client/src/helpers/constants.js
View File

@ -211,9 +211,14 @@ export const FILTERED = 'Filtered';
export const NOT_FILTERED = 'NotFiltered';
export const DISABLED_STATS_INTERVAL = 0;
export const STATS_INTERVALS_DAYS = [1, 7, 30, 90];
export const QUERY_LOG_INTERVALS_DAYS = [0.25, 1, 7, 30, 90];
export const HOUR = 60 * 60 * 1000;
export const DAY = HOUR * 24;
export const STATS_INTERVALS_DAYS = [DAY, DAY * 7, DAY * 30, DAY * 90];
export const QUERY_LOG_INTERVALS_DAYS = [HOUR * 6, DAY, DAY * 7, DAY * 30, DAY * 90];
export const FILTERS_INTERVALS_HOURS = [0, 1, 12, 24, 72, 168];

4
client/src/reducers/stats.js
View File

@ -25,7 +25,7 @@ const stats = handleActions(
[actions.getStatsConfigFailure]: (state) => ({ ...state, processingGetConfig: false }),
[actions.getStatsConfigSuccess]: (state, { payload }) => ({
...state,
interval: payload.interval,
...payload,
processingGetConfig: false,
}),
@ -33,7 +33,7 @@ const stats = handleActions(
[actions.setStatsConfigFailure]: (state) => ({ ...state, processingSetConfig: false }),
[actions.setStatsConfigSuccess]: (state, { payload }) => ({
...state,
interval: payload.interval,
...payload,
processingSetConfig: false,
}),

29
internal/aghnet/hostgen.go
View File

@ -1,8 +1,13 @@
package aghnet
import (
"fmt"
"net"
"strconv"
"strings"
"github.com/AdguardTeam/golibs/errors"
"github.com/AdguardTeam/golibs/stringutil"
)
// The maximum lengths of generated hostnames for different IP versions.
@ -59,3 +64,27 @@ func GenerateHostname(ip net.IP) (hostname string) {
return generateIPv6Hostname(ip)
}
// NewDomainNameSet returns nil and error, if list has duplicate or empty
// domain name. Otherwise returns a set, which contains non-FQDN domain names,
// and nil error.
func NewDomainNameSet(list []string) (set *stringutil.Set, err error) {
set = stringutil.NewSet()
for i, v := range list {
host := strings.ToLower(strings.TrimSuffix(v, "."))
// TODO(a.garipov): Think about ignoring empty (".") names in the
// future.
if host == "" {
return nil, errors.Error("host name is empty")
}
if set.Has(host) {
return nil, fmt.Errorf("duplicate host name %q at index %d", host, i)
}
set.Add(host)
}
return set, nil
}

38
internal/home/config.go
View File

@ -228,34 +228,32 @@ type tlsConfigSettings struct {
}
type queryLogConfig struct {
// Ignored is the list of host names, which should not be written to log.
Ignored []string `yaml:"ignored"`
// Interval is the interval for query log's files rotation.
Interval timeutil.Duration `yaml:"interval"`
// MemSize is the number of entries kept in memory before they are flushed
// to disk.
MemSize uint32 `yaml:"size_memory"`
// Enabled defines if the query log is enabled.
Enabled bool `yaml:"enabled"`
// FileEnabled defines, if the query log is written to the file.
FileEnabled bool `yaml:"file_enabled"`
// Interval is the interval for query log's files rotation.
Interval timeutil.Duration `yaml:"interval"`
// MemSize is the number of entries kept in memory before they are
// flushed to disk.
MemSize uint32 `yaml:"size_memory"`
// Ignored is the list of host names, which should not be written to
// log.
Ignored []string `yaml:"ignored"`
}
type statsConfig struct {
// Enabled defines if the statistics are enabled.
Enabled bool `yaml:"enabled"`
// Interval is the time interval for flushing statistics to the disk in
// days.
Interval uint32 `yaml:"interval"`
// Ignored is the list of host names, which should not be counted.
Ignored []string `yaml:"ignored"`
// Interval is the retention interval for statistics.
Interval timeutil.Duration `yaml:"interval"`
// Enabled defines if the statistics are enabled.
Enabled bool `yaml:"enabled"`
}
// config is the global configuration structure.
@ -322,7 +320,7 @@ var config = &configuration{
},
Stats: statsConfig{
Enabled: true,
Interval: 1,
Interval: timeutil.Duration{Duration: 1 * timeutil.Day},
Ignored: []string{},
},
// NOTE: Keep these parameters in sync with the one put into
@ -503,7 +501,7 @@ func (c *configuration) write() (err error) {
if Context.stats != nil {
statsConf := stats.Config{}
Context.stats.WriteDiskConfig(&statsConf)
config.Stats.Interval = statsConf.LimitDays
config.Stats.Interval = timeutil.Duration{Duration: statsConf.Limit}
config.Stats.Enabled = statsConf.Enabled
config.Stats.Ignored = statsConf.Ignored.Values()
slices.Sort(config.Stats.Ignored)

37
internal/home/dns.go
View File

@ -8,7 +8,6 @@ import (
"net/url"
"os"
"path/filepath"
"strings"
"github.com/AdguardTeam/AdGuardHome/internal/aghalg"
"github.com/AdguardTeam/AdGuardHome/internal/aghhttp"
@ -22,7 +21,6 @@ import (
"github.com/AdguardTeam/golibs/errors"
"github.com/AdguardTeam/golibs/log"
"github.com/AdguardTeam/golibs/netutil"
"github.com/AdguardTeam/golibs/stringutil"
"github.com/ameshkov/dnscrypt/v2"
yaml "gopkg.in/yaml.v3"
)
@ -54,13 +52,13 @@ func initDNS() (err error) {
statsConf := stats.Config{
Filename: filepath.Join(baseDir, "stats.db"),
LimitDays: config.Stats.Interval,
Limit: config.Stats.Interval.Duration,
ConfigModified: onConfigModified,
HTTPRegister: httpRegister,
Enabled: config.Stats.Enabled,
}
set, err := nonDupEmptyHostNames(config.Stats.Ignored)
set, err := aghnet.NewDomainNameSet(config.Stats.Ignored)
if err != nil {
return fmt.Errorf("statistics: ignored list: %w", err)
}
@ -84,13 +82,16 @@ func initDNS() (err error) {
FileEnabled: config.QueryLog.FileEnabled,
}
set, err = nonDupEmptyHostNames(config.QueryLog.Ignored)
set, err = aghnet.NewDomainNameSet(config.QueryLog.Ignored)
if err != nil {
return fmt.Errorf("querylog: ignored list: %w", err)
}
conf.Ignored = set
Context.queryLog = querylog.New(conf)
Context.queryLog, err = querylog.New(conf)
if err != nil {
return fmt.Errorf("init querylog: %w", err)
}
Context.filters, err = filtering.New(config.DNS.DnsfilterConf, nil)
if err != nil {
@ -535,30 +536,6 @@ func closeDNSServer() {
log.Debug("all dns modules are closed")
}
// nonDupEmptyHostNames returns nil and error, if list has duplicate or empty
// host name. Otherwise returns a set, which contains lowercase host names
// without dot at the end, and nil error.
func nonDupEmptyHostNames(list []string) (set *stringutil.Set, err error) {
set = stringutil.NewSet()
for _, v := range list {
host := strings.ToLower(strings.TrimSuffix(v, "."))
// TODO(a.garipov): Think about ignoring empty (".") names in
// the future.
if host == "" {
return nil, errors.Error("host name is empty")
}
if set.Has(host) {
return nil, fmt.Errorf("duplicate host name %q", host)
}
set.Add(host)
}
return set, nil
}
// safeSearchResolver is a [filtering.Resolver] implementation used for safe
// search.
type safeSearchResolver struct{}

44
internal/home/upgrade.go
View File

@ -22,7 +22,7 @@ import (
)
// currentSchemaVersion is the current schema version.
const currentSchemaVersion = 19
const currentSchemaVersion = 20
// These aliases are provided for convenience.
type (
@ -92,6 +92,7 @@ func upgradeConfigSchema(oldVersion int, diskConf yobj) (err error) {
upgradeSchema16to17,
upgradeSchema17to18,
upgradeSchema18to19,
upgradeSchema19to20,
}
n := 0
@ -1064,6 +1065,47 @@ func upgradeSchema18to19(diskConf yobj) (err error) {
return nil
}
// upgradeSchema19to20 performs the following changes:
//
// # BEFORE:
// 'statistics':
// 'interval': 1
//
// # AFTER:
// 'statistics':
// 'interval': 24h
func upgradeSchema19to20(diskConf yobj) (err error) {
log.Printf("Upgrade yaml: 19 to 20")
diskConf["schema_version"] = 20
statsVal, ok := diskConf["statistics"]
if !ok {
return nil
}
var stats yobj
stats, ok = statsVal.(yobj)
if !ok {
return fmt.Errorf("unexpected type of stats: %T", statsVal)
}
const field = "interval"
// Set the initial value from the global configuration structure.
statsIvl := 1
statsIvlVal, ok := stats[field]
if ok {
statsIvl, ok = statsIvlVal.(int)
if !ok {
return fmt.Errorf("unexpected type of %s: %T", field, statsIvlVal)
}
}
stats[field] = timeutil.Duration{Duration: time.Duration(statsIvl) * timeutil.Day}
return nil
}
// TODO(a.garipov): Replace with log.Output when we port it to our logging
// package.
func funcName() string {

95
internal/home/upgrade_test.go
View File

@ -951,3 +951,98 @@ func TestUpgradeSchema18to19(t *testing.T) {
})
}
}
func TestUpgradeSchema19to20(t *testing.T) {
testCases := []struct {
ivl any
want any
wantErr string
name string
}{{
ivl: 1,
want: timeutil.Duration{Duration: timeutil.Day},
wantErr: "",
name: "success",
}, {
ivl: 0.25,
want: 0,
wantErr: "unexpected type of interval: float64",
name: "fail",
}}
for _, tc := range testCases {
conf := yobj{
"statistics": yobj{
"interval": tc.ivl,
},
"schema_version": 19,
}
t.Run(tc.name, func(t *testing.T) {
err := upgradeSchema19to20(conf)
if tc.wantErr != "" {
require.Error(t, err)
assert.Equal(t, tc.wantErr, err.Error())
return
}
require.NoError(t, err)
require.Equal(t, conf["schema_version"], 20)
statsVal, ok := conf["statistics"]
require.True(t, ok)
var stats yobj
stats, ok = statsVal.(yobj)
require.True(t, ok)
var newIvl timeutil.Duration
newIvl, ok = stats["interval"].(timeutil.Duration)
require.True(t, ok)
assert.Equal(t, tc.want, newIvl)
})
}
t.Run("no_stats", func(t *testing.T) {
err := upgradeSchema19to20(yobj{})
assert.NoError(t, err)
})
t.Run("bad_stats", func(t *testing.T) {
err := upgradeSchema19to20(yobj{
"statistics": 0,
})
testutil.AssertErrorMsg(t, "unexpected type of stats: int", err)
})
t.Run("no_field", func(t *testing.T) {
conf := yobj{
"statistics": yobj{},
}
err := upgradeSchema19to20(conf)
require.NoError(t, err)
statsVal, ok := conf["statistics"]
require.True(t, ok)
var stats yobj
stats, ok = statsVal.(yobj)
require.True(t, ok)
var ivl any
ivl, ok = stats["interval"]
require.True(t, ok)
var ivlVal timeutil.Duration
ivlVal, ok = ivl.(timeutil.Duration)
require.True(t, ok)
assert.Equal(t, 24*time.Hour, ivlVal.Duration)
})
}

131
internal/querylog/http.go
View File

@ -13,9 +13,11 @@ import (
"github.com/AdguardTeam/AdGuardHome/internal/aghalg"
"github.com/AdguardTeam/AdGuardHome/internal/aghhttp"
"github.com/AdguardTeam/AdGuardHome/internal/aghnet"
"github.com/AdguardTeam/golibs/log"
"github.com/AdguardTeam/golibs/stringutil"
"github.com/AdguardTeam/golibs/timeutil"
"golang.org/x/exp/slices"
"golang.org/x/net/idna"
)
@ -25,8 +27,8 @@ type configJSON struct {
// fractional numbers and not mess the API users by changing the units.
Interval float64 `json:"interval"`
// Enabled shows if the querylog is enabled. It is an [aghalg.NullBool]
// to be able to tell when it's set without using pointers.
// Enabled shows if the querylog is enabled. It is an aghalg.NullBool to
// be able to tell when it's set without using pointers.
Enabled aghalg.NullBool `json:"enabled"`
// AnonymizeClientIP shows if the clients' IP addresses must be anonymized.
@ -35,12 +37,39 @@ type configJSON struct {
AnonymizeClientIP aghalg.NullBool `json:"anonymize_client_ip"`
}
// getConfigResp is the JSON structure for the querylog configuration.
type getConfigResp struct {
// Ignored is the list of host names, which should not be written to log.
Ignored []string `json:"ignored"`
// Interval is the querylog rotation interval in milliseconds.
Interval float64 `json:"interval"`
// Enabled shows if the querylog is enabled. It is an aghalg.NullBool to
// be able to tell when it's set without using pointers.
Enabled aghalg.NullBool `json:"enabled"`
// AnonymizeClientIP shows if the clients' IP addresses must be anonymized.
// It is an aghalg.NullBool to be able to tell when it's set without using
// pointers.
//
// TODO(a.garipov): Consider using separate setting for statistics.
AnonymizeClientIP aghalg.NullBool `json:"anonymize_client_ip"`
}
// Register web handlers
func (l *queryLog) initWeb() {
l.conf.HTTPRegister(http.MethodGet, "/control/querylog", l.handleQueryLog)
l.conf.HTTPRegister(http.MethodGet, "/control/querylog_info", l.handleQueryLogInfo)
l.conf.HTTPRegister(http.MethodPost, "/control/querylog_clear", l.handleQueryLogClear)
l.conf.HTTPRegister(http.MethodPost, "/control/querylog_config", l.handleQueryLogConfig)
l.conf.HTTPRegister(http.MethodGet, "/control/querylog/config", l.handleGetQueryLogConfig)
l.conf.HTTPRegister(
http.MethodPut,
"/control/querylog/config/update",
l.handlePutQueryLogConfig,
)
}
func (l *queryLog) handleQueryLog(w http.ResponseWriter, r *http.Request) {
@ -64,11 +93,41 @@ func (l *queryLog) handleQueryLogClear(_ http.ResponseWriter, _ *http.Request) {
l.clear()
}
// Get configuration
// handleQueryLogInfo handles requests to the GET /control/querylog_info
// endpoint.
//
// Deprecated: Remove it when migration to the new API is over.
func (l *queryLog) handleQueryLogInfo(w http.ResponseWriter, r *http.Request) {
l.lock.Lock()
defer l.lock.Unlock()
ivl := l.conf.RotationIvl
if !checkInterval(ivl) {
// NOTE: If interval is custom we set it to 90 days for compatibility
// with old API.
ivl = timeutil.Day * 90
}
_ = aghhttp.WriteJSONResponse(w, r, configJSON{
Enabled: aghalg.BoolToNullBool(l.conf.Enabled),
Interval: l.conf.RotationIvl.Hours() / 24,
Interval: ivl.Hours() / 24,
AnonymizeClientIP: aghalg.BoolToNullBool(l.conf.AnonymizeClientIP),
})
}
// handleGetQueryLogConfig handles requests to the GET /control/querylog/config
// endpoint.
func (l *queryLog) handleGetQueryLogConfig(w http.ResponseWriter, r *http.Request) {
l.lock.Lock()
defer l.lock.Unlock()
ignored := l.conf.Ignored.Values()
slices.Sort(ignored)
_ = aghhttp.WriteJSONResponse(w, r, getConfigResp{
Ignored: ignored,
Interval: float64(l.conf.RotationIvl.Milliseconds()),
Enabled: aghalg.BoolToNullBool(l.conf.Enabled),
AnonymizeClientIP: aghalg.BoolToNullBool(l.conf.AnonymizeClientIP),
})
}
@ -88,6 +147,8 @@ func AnonymizeIP(ip net.IP) {
}
// handleQueryLogConfig handles the POST /control/querylog_config queries.
//
// Deprecated: Remove it when migration to the new API is over.
func (l *queryLog) handleQueryLogConfig(w http.ResponseWriter, r *http.Request) {
// Set NaN as initial value to be able to know if it changed later by
// comparing it to NaN.
@ -103,6 +164,7 @@ func (l *queryLog) handleQueryLogConfig(w http.ResponseWriter, r *http.Request)
}
ivl := time.Duration(float64(timeutil.Day) * newConf.Interval)
hasIvl := !math.IsNaN(newConf.Interval)
if hasIvl && !checkInterval(ivl) {
aghhttp.Error(r, w, http.StatusBadRequest, "unsupported interval")
@ -115,8 +177,6 @@ func (l *queryLog) handleQueryLogConfig(w http.ResponseWriter, r *http.Request)
l.lock.Lock()
defer l.lock.Unlock()
// Copy data, modify it, then activate. Other threads (readers) don't need
// to use this lock.
conf := *l.conf
if newConf.Enabled != aghalg.NBNull {
conf.Enabled = newConf.Enabled == aghalg.NBTrue
@ -138,6 +198,65 @@ func (l *queryLog) handleQueryLogConfig(w http.ResponseWriter, r *http.Request)
l.conf = &conf
}
// handlePutQueryLogConfig handles the PUT /control/querylog/config/update
// queries.
func (l *queryLog) handlePutQueryLogConfig(w http.ResponseWriter, r *http.Request) {
newConf := &getConfigResp{}
err := json.NewDecoder(r.Body).Decode(newConf)
if err != nil {
aghhttp.Error(r, w, http.StatusBadRequest, "%s", err)
return
}
set, err := aghnet.NewDomainNameSet(newConf.Ignored)
if err != nil {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "ignored: %s", err)
return
}
ivl := time.Duration(newConf.Interval) * time.Millisecond
err = validateIvl(ivl)
if err != nil {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "unsupported interval: %s", err)
return
}
if newConf.Enabled == aghalg.NBNull {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "enabled is null")
return
}
if newConf.AnonymizeClientIP == aghalg.NBNull {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "anonymize_client_ip is null")
return
}
defer l.conf.ConfigModified()
l.lock.Lock()
defer l.lock.Unlock()
conf := *l.conf
conf.Ignored = set
conf.RotationIvl = ivl
conf.Enabled = newConf.Enabled == aghalg.NBTrue
conf.AnonymizeClientIP = newConf.AnonymizeClientIP == aghalg.NBTrue
if conf.AnonymizeClientIP {
l.anonymizer.Store(AnonymizeIP)
} else {
l.anonymizer.Store(nil)
}
l.conf = &conf
}
// "value" -> value, return TRUE
func getDoubleQuotesEnclosedValue(s *string) bool {
t := *s

17
internal/querylog/qlog.go
View File

@ -132,6 +132,20 @@ func checkInterval(ivl time.Duration) (ok bool) {
return ivl == quarterDay || ivl == day || ivl == week || ivl == month || ivl == threeMonths
}
// validateIvl returns an error if ivl is less than an hour or more than a
// year.
func validateIvl(ivl time.Duration) (err error) {
if ivl < time.Hour {
return errors.Error("less than an hour")
}
if ivl > timeutil.Day*365 {
return errors.Error("more than a year")
}
return nil
}
func (l *queryLog) WriteDiskConfig(c *Config) {
*c = *l.conf
}
@ -258,6 +272,9 @@ func (l *queryLog) Add(params *AddParams) {
// ShouldLog returns true if request for the host should be logged.
func (l *queryLog) ShouldLog(host string, _, _ uint16) bool {
l.lock.Lock()
defer l.lock.Unlock()
return !l.isIgnored(host)
}

15
internal/querylog/qlog_test.go
View File

@ -22,13 +22,14 @@ func TestMain(m *testing.M) {
// TestQueryLog tests adding and loading (with filtering) entries from disk and
// memory.
func TestQueryLog(t *testing.T) {
l := newQueryLog(Config{
l, err := newQueryLog(Config{
Enabled: true,
FileEnabled: true,
RotationIvl: timeutil.Day,
MemSize: 100,
BaseDir: t.TempDir(),
})
require.NoError(t, err)
// Add disk entries.
addEntry(l, "example.org", net.IPv4(1, 1, 1, 1), net.IPv4(2, 2, 2, 1))
@ -125,12 +126,13 @@ func TestQueryLog(t *testing.T) {
}
func TestQueryLogOffsetLimit(t *testing.T) {
l := newQueryLog(Config{
l, err := newQueryLog(Config{
Enabled: true,
RotationIvl: timeutil.Day,
MemSize: 100,
BaseDir: t.TempDir(),
})
require.NoError(t, err)
const (
entNum = 10
@ -199,13 +201,14 @@ func TestQueryLogOffsetLimit(t *testing.T) {
}
func TestQueryLogMaxFileScanEntries(t *testing.T) {
l := newQueryLog(Config{
l, err := newQueryLog(Config{
Enabled: true,
FileEnabled: true,
RotationIvl: timeutil.Day,
MemSize: 100,
BaseDir: t.TempDir(),
})
require.NoError(t, err)
const entNum = 10
// Add entries to the log.
@ -227,13 +230,14 @@ func TestQueryLogMaxFileScanEntries(t *testing.T) {
}
func TestQueryLogFileDisabled(t *testing.T) {
l := newQueryLog(Config{
l, err := newQueryLog(Config{
Enabled: true,
FileEnabled: false,
RotationIvl: timeutil.Day,
MemSize: 2,
BaseDir: t.TempDir(),
})
require.NoError(t, err)
addEntry(l, "example1.org", net.IPv4(1, 1, 1, 1), net.IPv4(2, 2, 2, 1))
addEntry(l, "example2.org", net.IPv4(1, 1, 1, 1), net.IPv4(2, 2, 2, 1))
@ -254,13 +258,14 @@ func TestQueryLogShouldLog(t *testing.T) {
)
set := stringutil.NewSet(ignored1, ignored2)
l := newQueryLog(Config{
l, err := newQueryLog(Config{
Enabled: true,
RotationIvl: timeutil.Day,
MemSize: 100,
BaseDir: t.TempDir(),
Ignored: set,
})
require.NoError(t, err)
testCases := []struct {
name string

18
internal/querylog/querylog.go
View File

@ -1,6 +1,7 @@
package querylog
import (
"fmt"
"net"
"path/filepath"
"time"
@ -9,9 +10,7 @@ import (
"github.com/AdguardTeam/AdGuardHome/internal/aghnet"
"github.com/AdguardTeam/AdGuardHome/internal/filtering"
"github.com/AdguardTeam/golibs/errors"
"github.com/AdguardTeam/golibs/log"
"github.com/AdguardTeam/golibs/stringutil"
"github.com/AdguardTeam/golibs/timeutil"
"github.com/miekg/dns"
)
@ -135,12 +134,12 @@ func (p *AddParams) validate() (err error) {
}
// New creates a new instance of the query log.
func New(conf Config) (ql QueryLog) {
func New(conf Config) (ql QueryLog, err error) {
return newQueryLog(conf)
}
// newQueryLog crates a new queryLog.
func newQueryLog(conf Config) (l *queryLog) {
func newQueryLog(conf Config) (l *queryLog, err error) {
findClient := conf.FindClient
if findClient == nil {
findClient = func(_ []string) (_ *Client, _ error) {
@ -158,13 +157,10 @@ func newQueryLog(conf Config) (l *queryLog) {
l.conf = &Config{}
*l.conf = conf
if !checkInterval(conf.RotationIvl) {
log.Info(
"querylog: warning: unsupported rotation interval %s, setting to 1 day",
conf.RotationIvl,
)
l.conf.RotationIvl = timeutil.Day
err = validateIvl(conf.RotationIvl)
if err != nil {
return nil, fmt.Errorf("unsupported interval: %w", err)
}
return l
return l, nil
}

3
internal/querylog/search_test.go
View File

@ -35,7 +35,7 @@ func TestQueryLog_Search_findClient(t *testing.T) {
return nil, nil
}
l := newQueryLog(Config{
l, err := newQueryLog(Config{
FindClient: findClient,
BaseDir: t.TempDir(),
RotationIvl: timeutil.Day,
@ -44,6 +44,7 @@ func TestQueryLog_Search_findClient(t *testing.T) {
FileEnabled: true,
AnonymizeClientIP: false,
})
require.NoError(t, err)
t.Cleanup(l.Close)
q := &dns.Msg{

104
internal/stats/http.go
View File

@ -7,8 +7,12 @@ import (
"net/http"
"time"
"github.com/AdguardTeam/AdGuardHome/internal/aghalg"
"github.com/AdguardTeam/AdGuardHome/internal/aghhttp"
"github.com/AdguardTeam/AdGuardHome/internal/aghnet"
"github.com/AdguardTeam/golibs/log"
"github.com/AdguardTeam/golibs/timeutil"
"golang.org/x/exp/slices"
)
// topAddrs is an alias for the types of the TopFoo fields of statsResponse.
@ -44,7 +48,7 @@ func (s *StatsCtx) handleStats(w http.ResponseWriter, r *http.Request) {
defer s.lock.Unlock()
start := time.Now()
resp, ok := s.getData(s.limitHours)
resp, ok := s.getData(uint32(s.limit.Hours()))
log.Debug("stats: prepared data in %v", time.Since(start))
if !ok {
@ -63,20 +67,62 @@ type configResp struct {
IntervalDays uint32 `json:"interval"`
}
// getConfigResp is the response to the GET /control/stats_info.
type getConfigResp struct {
// Ignored is the list of host names, which should not be counted.
Ignored []string `json:"ignored"`
// Interval is the statistics rotation interval in milliseconds.
Interval float64 `json:"interval"`
// Enabled shows if statistics are enabled. It is an aghalg.NullBool to be
// able to tell when it's set without using pointers.
Enabled aghalg.NullBool `json:"enabled"`
}
// handleStatsInfo handles requests to the GET /control/stats_info endpoint.
//
// Deprecated: Remove it when migration to the new API is over.
func (s *StatsCtx) handleStatsInfo(w http.ResponseWriter, r *http.Request) {
s.lock.Lock()
defer s.lock.Unlock()
resp := configResp{IntervalDays: s.limitHours / 24}
days := uint32(s.limit / timeutil.Day)
ok := checkInterval(days)
if !ok || (s.enabled && days == 0) {
// NOTE: If interval is custom we set it to 90 days for compatibility
// with old API.
days = 90
}
resp := configResp{IntervalDays: days}
if !s.enabled {
resp.IntervalDays = 0
}
_ = aghhttp.WriteJSONResponse(w, r, resp)
}
// handleGetStatsConfig handles requests to the GET /control/stats/config
// endpoint.
func (s *StatsCtx) handleGetStatsConfig(w http.ResponseWriter, r *http.Request) {
s.lock.Lock()
defer s.lock.Unlock()
ignored := s.ignored.Values()
slices.Sort(ignored)
resp := getConfigResp{
Ignored: ignored,
Interval: float64(s.limit.Milliseconds()),
Enabled: aghalg.BoolToNullBool(s.enabled),
}
_ = aghhttp.WriteJSONResponse(w, r, resp)
}
// handleStatsConfig handles requests to the POST /control/stats_config
// endpoint.
//
// Deprecated: Remove it when migration to the new API is over.
func (s *StatsCtx) handleStatsConfig(w http.ResponseWriter, r *http.Request) {
reqData := configResp{}
err := json.NewDecoder(r.Body).Decode(&reqData)
@ -92,8 +138,55 @@ func (s *StatsCtx) handleStatsConfig(w http.ResponseWriter, r *http.Request) {
return
}
s.setLimit(int(reqData.IntervalDays))
s.configModified()
defer s.configModified()
s.lock.Lock()
defer s.lock.Unlock()
limit := time.Duration(reqData.IntervalDays) * timeutil.Day
s.setLimit(limit)
}
// handlePutStatsConfig handles requests to the PUT /control/stats/config/update
// endpoint.
func (s *StatsCtx) handlePutStatsConfig(w http.ResponseWriter, r *http.Request) {
reqData := getConfigResp{}
err := json.NewDecoder(r.Body).Decode(&reqData)
if err != nil {
aghhttp.Error(r, w, http.StatusBadRequest, "json decode: %s", err)
return
}
set, err := aghnet.NewDomainNameSet(reqData.Ignored)
if err != nil {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "ignored: %s", err)
return
}
ivl := time.Duration(reqData.Interval) * time.Millisecond
err = validateIvl(ivl)
if err != nil {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "unsupported interval: %s", err)
return
}
if reqData.Enabled == aghalg.NBNull {
aghhttp.Error(r, w, http.StatusUnprocessableEntity, "enabled is null")
return
}
defer s.configModified()
s.lock.Lock()
defer s.lock.Unlock()
s.ignored = set
s.limit = ivl
s.enabled = reqData.Enabled == aghalg.NBTrue
}
// handleStatsReset handles requests to the POST /control/stats_reset endpoint.
@ -114,4 +207,7 @@ func (s *StatsCtx) initWeb() {
s.httpRegister(http.MethodPost, "/control/stats_reset", s.handleStatsReset)
s.httpRegister(http.MethodPost, "/control/stats_config", s.handleStatsConfig)
s.httpRegister(http.MethodGet, "/control/stats_info", s.handleStatsInfo)
s.httpRegister(http.MethodGet, "/control/stats/config", s.handleGetStatsConfig)
s.httpRegister(http.MethodPut, "/control/stats/config/update", s.handlePutStatsConfig)
}

152
internal/stats/http_test.go Normal file
View File

@ -0,0 +1,152 @@
package stats
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"time"
"github.com/AdguardTeam/AdGuardHome/internal/aghalg"
"github.com/AdguardTeam/golibs/testutil"
"github.com/AdguardTeam/golibs/timeutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestHandleStatsConfig(t *testing.T) {
const (
smallIvl = 1 * time.Minute
minIvl = 1 * time.Hour
maxIvl = 365 * timeutil.Day
)
conf := Config{
Filename: filepath.Join(t.TempDir(), "stats.db"),
Limit: time.Hour * 24,
Enabled: true,
UnitID: func() (id uint32) { return 0 },
ConfigModified: func() {},
}
testCases := []struct {
name string
body getConfigResp
wantCode int
wantErr string
}{{
name: "set_ivl_1_minIvl",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(minIvl.Milliseconds()),
Ignored: []string{},
},
wantCode: http.StatusOK,
wantErr: "",
}, {
name: "small_interval",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(smallIvl.Milliseconds()),
Ignored: []string{},
},
wantCode: http.StatusUnprocessableEntity,
wantErr: "unsupported interval: less than an hour\n",
}, {
name: "big_interval",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(maxIvl.Milliseconds() + minIvl.Milliseconds()),
Ignored: []string{},
},
wantCode: http.StatusUnprocessableEntity,
wantErr: "unsupported interval: more than a year\n",
}, {
name: "set_ignored_ivl_1_maxIvl",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(maxIvl.Milliseconds()),
Ignored: []string{
"ignor.ed",
},
},
wantCode: http.StatusOK,
wantErr: "",
}, {
name: "ignored_duplicate",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(minIvl.Milliseconds()),
Ignored: []string{
"ignor.ed",
"ignor.ed",
},
},
wantCode: http.StatusUnprocessableEntity,
wantErr: "ignored: duplicate host name \"ignor.ed\" at index 1\n",
}, {
name: "ignored_empty",
body: getConfigResp{
Enabled: aghalg.NBTrue,
Interval: float64(minIvl.Milliseconds()),
Ignored: []string{
"",
},
},
wantCode: http.StatusUnprocessableEntity,
wantErr: "ignored: host name is empty\n",
}, {
name: "enabled_is_null",
body: getConfigResp{
Enabled: aghalg.NBNull,
Interval: float64(minIvl.Milliseconds()),
Ignored: []string{},
},
wantCode: http.StatusUnprocessableEntity,
wantErr: "enabled is null\n",
}}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
s, err := New(conf)
require.NoError(t, err)
s.Start()
testutil.CleanupAndRequireSuccess(t, s.Close)
buf, err := json.Marshal(tc.body)
require.NoError(t, err)
const (
configGet = "/control/stats/config"
configPut = "/control/stats/config/update"
)
req := httptest.NewRequest(http.MethodPut, configPut, bytes.NewReader(buf))
rw := httptest.NewRecorder()
s.handlePutStatsConfig(rw, req)
require.Equal(t, tc.wantCode, rw.Code)
if tc.wantCode != http.StatusOK {
assert.Equal(t, tc.wantErr, rw.Body.String())
return
}
resp := httptest.NewRequest(http.MethodGet, configGet, nil)
rw = httptest.NewRecorder()
s.handleGetStatsConfig(rw, resp)
require.Equal(t, http.StatusOK, rw.Code)
ans := getConfigResp{}
err = json.Unmarshal(rw.Body.Bytes(), &ans)
require.NoError(t, err)
assert.Equal(t, tc.body, ans)
})
}
}

60
internal/stats/stats.go
View File

@ -16,6 +16,7 @@ import (
"github.com/AdguardTeam/golibs/errors"
"github.com/AdguardTeam/golibs/log"
"github.com/AdguardTeam/golibs/stringutil"
"github.com/AdguardTeam/golibs/timeutil"
"go.etcd.io/bbolt"
)
@ -25,6 +26,20 @@ func checkInterval(days uint32) (ok bool) {
return days == 0 || days == 1 || days == 7 || days == 30 || days == 90
}
// validateIvl returns an error if ivl is less than an hour or more than a
// year.
func validateIvl(ivl time.Duration) (err error) {
if ivl < time.Hour {
return errors.Error("less than an hour")
}
if ivl > timeutil.Day*365 {
return errors.Error("more than a year")
}
return nil
}
// Config is the configuration structure for the statistics collecting.
type Config struct {
// UnitID is the function to generate the identifier for current unit. If
@ -42,9 +57,8 @@ type Config struct {
// Filename is the name of the database file.
Filename string
// LimitDays is the maximum number of days to collect statistics into the
// current unit.
LimitDays uint32
// Limit is an upper limit for collecting statistics.
Limit time.Duration
// Enabled tells if the statistics are enabled.
Enabled bool
@ -105,11 +119,8 @@ type StatsCtx struct {
// enabled tells if the statistics are enabled.
enabled bool
// limitHours is the maximum number of hours to collect statistics into the
// current unit.
//
// TODO(s.chzhen): Rewrite to use time.Duration.
limitHours uint32
// limit is an upper limit for collecting statistics.
limit time.Duration
// ignored is the list of host names, which should not be counted.
ignored *stringutil.Set
@ -128,9 +139,14 @@ func New(conf Config) (s *StatsCtx, err error) {
httpRegister: conf.HTTPRegister,
ignored: conf.Ignored,
}
if s.limitHours = conf.LimitDays * 24; !checkInterval(conf.LimitDays) {
s.limitHours = 24
err = validateIvl(conf.Limit)
if err != nil {
return nil, fmt.Errorf("unsupported interval: %w", err)
}
s.limit = conf.Limit
if s.unitIDGen = newUnitID; conf.UnitID != nil {
s.unitIDGen = conf.UnitID
}
@ -150,7 +166,7 @@ func New(conf Config) (s *StatsCtx, err error) {
return nil, fmt.Errorf("stats: opening a transaction: %w", err)
}
deleted := deleteOldUnits(tx, id-s.limitHours-1)
deleted := deleteOldUnits(tx, id-uint32(s.limit.Hours())-1)
udb = loadUnitFromDB(tx, id)
err = finishTxn(tx, deleted > 0)
@ -231,7 +247,7 @@ func (s *StatsCtx) Update(e Entry) {
s.lock.Lock()
defer s.lock.Unlock()
if !s.enabled || s.limitHours == 0 {
if !s.enabled || s.limit == 0 {
return
}
@ -263,7 +279,7 @@ func (s *StatsCtx) WriteDiskConfig(dc *Config) {
s.lock.Lock()
defer s.lock.Unlock()
dc.LimitDays = s.limitHours / 24
dc.Limit = s.limit
dc.Enabled = s.enabled
dc.Ignored = s.ignored
}
@ -273,7 +289,7 @@ func (s *StatsCtx) TopClientsIP(maxCount uint) (ips []netip.Addr) {
s.lock.Lock()
defer s.lock.Unlock()
limit := s.limitHours
limit := uint32(s.limit.Hours())
if !s.enabled || limit == 0 {
return nil
}
@ -377,7 +393,7 @@ func (s *StatsCtx) flush() (cont bool, sleepFor time.Duration) {
return false, 0
}
limit := s.limitHours
limit := uint32(s.limit.Hours())
if limit == 0 || ptr.id == id {
return true, time.Second
}
@ -436,14 +452,14 @@ func (s *StatsCtx) periodicFlush() {
log.Debug("periodic flushing finished")
}
func (s *StatsCtx) setLimit(limitDays int) {
s.lock.Lock()
defer s.lock.Unlock()
if limitDays != 0 {
// setLimit sets the limit. s.lock is expected to be locked.
//
// TODO(s.chzhen): Remove it when migration to the new API is over.
func (s *StatsCtx) setLimit(limit time.Duration) {
if limit != 0 {
s.enabled = true
s.limitHours = uint32(24 * limitDays)
log.Debug("stats: set limit: %d days", limitDays)
s.limit = limit
log.Debug("stats: set limit: %d days", limit/timeutil.Day)
return
}

7
internal/stats/stats_internal_test.go
View File

@ -9,6 +9,7 @@ import (
"time"
"github.com/AdguardTeam/golibs/testutil"
"github.com/AdguardTeam/golibs/timeutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@ -35,9 +36,9 @@ func TestStats_races(t *testing.T) {
var r uint32
idGen := func() (id uint32) { return atomic.LoadUint32(&r) }
conf := Config{
UnitID: idGen,
Filename: filepath.Join(t.TempDir(), "./stats.db"),
LimitDays: 1,
UnitID: idGen,
Filename: filepath.Join(t.TempDir(), "./stats.db"),
Limit: timeutil.Day,
}
s, err := New(conf)

11
internal/stats/stats_test.go
View File

@ -13,6 +13,7 @@ import (
"github.com/AdguardTeam/AdGuardHome/internal/stats"
"github.com/AdguardTeam/golibs/netutil"
"github.com/AdguardTeam/golibs/testutil"
"github.com/AdguardTeam/golibs/timeutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@ -51,10 +52,10 @@ func TestStats(t *testing.T) {
handlers := map[string]http.Handler{}
conf := stats.Config{
Filename: filepath.Join(t.TempDir(), "stats.db"),
LimitDays: 1,
Enabled: true,
UnitID: constUnitID,
Filename: filepath.Join(t.TempDir(), "stats.db"),
Limit: timeutil.Day,
Enabled: true,
UnitID: constUnitID,
HTTPRegister: func(_, url string, handler http.HandlerFunc) {
handlers[url] = handler
},
@ -158,7 +159,7 @@ func TestLargeNumbers(t *testing.T) {
conf := stats.Config{
Filename: filepath.Join(t.TempDir(), "stats.db"),
LimitDays: 1,
Limit: timeutil.Day,
Enabled: true,
UnitID: func() (id uint32) { return atomic.LoadUint32(&curHour) },
HTTPRegister: func(_, url string, handler http.HandlerFunc) { handlers[url] = handler },

65
openapi/CHANGELOG.md
View File

@ -18,6 +18,71 @@
## v0.107.27: API changes
### Deprecated statistics APIs
* The `GET /control/stats_info` HTTP API; use the new `GET
/control/stats/config` API instead.
**NOTE:** If `interval` was configured by editing configuration file or new
HTTP API call `PUT /control/stats/config/update` and it's not equal to
previous allowed enum values then it will be equal to `90` days for
compatibility reasons.
* The `POST /control/stats_config` HTTP API; use the new `PUT
/control/stats/config/update` API instead.
### New statistics APIs
* The new `GET /control/stats/config` HTTP API.
* The new `PUT /control/stats/config/update` HTTP API allows config updates.
These `control/stats/config/update` and `control/stats/config` APIs accept and
return a JSON object with the following format:
```json
{
"enabled": true,
"interval": 3600,
"ignored": ["example.com"],
}
```
### Deprecated query log APIs
* The `GET /control/querylog_info` HTTP API; use the new `GET
/control/querylog/config` API instead.
**NOTE:** If `interval` was configured by editing configuration file or new
HTTP API call `PUT /control/querylog/config/update` and it's not equal to
previous allowed enum values then it will be equal to `90` days for
compatibility reasons.
* The `POST /control/querylog_config` HTTP API; use the new `PUT
/control/querylog/config/update` API instead.
### New query log APIs
* The new `GET /control/querylog/config` HTTP API.
* The new `PUT /control/querylog/config/update` HTTP API allows config updates.
These `control/querylog/config/update` and `control/querylog/config` APIs
accept and return a JSON object with the following format:
```json
{
"enabled": true,
"anonymize_client_ip": false,
"interval": 3600,
"ignored": ["example.com"],
}
```
## v0.107.23: API changes
### Experimental “beta” APIs removed

125
openapi/openapi.yaml
View File

@ -226,6 +226,14 @@
'$ref': '#/components/schemas/QueryLog'
'/querylog_info':
'get':
'deprecated': true
'description': |
Deprecated: Use `GET /querylog/config` instead.
NOTE: If `interval` was configured by editing configuration file or new
HTTP API call `PUT /querylog/config/update` and it's not equal to
previous allowed enum values then it will be equal to `90` days for
compatibility reasons.
'tags':
- 'log'
'operationId': 'queryLogInfo'
@ -239,6 +247,9 @@
'$ref': '#/components/schemas/QueryLogConfig'
'/querylog_config':
'post':
'deprecated': true
'description': >
Deprecated: Use `PUT /querylog/config/update` instead.
'tags':
- 'log'
'operationId': 'queryLogConfig'
@ -260,6 +271,34 @@
'responses':
'200':
'description': 'OK.'
'/querylog/config':
'get':
'tags':
- 'log'
'operationId': 'getQueryLogConfig'
'summary': 'Get query log parameters'
'responses':
'200':
'description': 'OK.'
'content':
'application/json':
'schema':
'$ref': '#/components/schemas/GetQueryLogConfigResponse'
'/querylog/config/update':
'put':
'tags':
- 'log'
'operationId': 'putQueryLogConfig'
'summary': 'Set query log parameters'
'requestBody':
'content':
'application/json':
'schema':
'$ref': '#/components/schemas/PutQueryLogConfigUpdateRequest'
'required': true
'responses':
'200':
'description': 'OK.'
'/stats':
'get':
'tags':
@ -284,6 +323,14 @@
'description': 'OK.'
'/stats_info':
'get':
'deprecated': true
'description': |
Deprecated: Use `GET /stats/config` instead.
NOTE: If `interval` was configured by editing configuration file or new
HTTP API call `PUT /stats/config/update` and it's not equal to
previous allowed enum values then it will be equal to `90` days for
compatibility reasons.
'tags':
- 'stats'
'operationId': 'statsInfo'
@ -297,6 +344,9 @@
'$ref': '#/components/schemas/StatsConfig'
'/stats_config':
'post':
'deprecated': true
'description': >
Deprecated: Use `PUT /stats/config/update` instead.
'tags':
- 'stats'
'operationId': 'statsConfig'
@ -309,6 +359,34 @@
'responses':
'200':
'description': 'OK.'
'/stats/config':
'get':
'tags':
- 'stats'
'operationId': 'getStatsConfig'
'summary': 'Get statistics parameters'
'responses':
'200':
'description': 'OK.'
'content':
'application/json':
'schema':
'$ref': '#/components/schemas/GetStatsConfigResponse'
'/stats/config/update':
'put':
'tags':
- 'stats'
'operationId': 'putStatsConfig'
'summary': 'Set statistics parameters'
'requestBody':
'content':
'application/json':
'schema':
'$ref': '#/components/schemas/PutStatsConfigUpdateRequest'
'required': true
'responses':
'200':
'description': 'OK.'
'/tls/status':
'get':
'tags':
@ -1656,6 +1734,27 @@
- 30
- 90
'type': 'integer'
'GetStatsConfigResponse':
'type': 'object'
'description': 'Statistics configuration'
'required':
- 'enabled'
- 'interval'
- 'ignored'
'properties':
'enabled':
'description': 'Are statistics enabled'
'type': 'boolean'
'interval':
'description': 'Statistics rotation interval'
'type': 'number'
'ignored':
'description': 'List of host names, which should not be counted'
'type': 'array'
'items':
'type': 'string'
'PutStatsConfigUpdateRequest':
'$ref': '#/components/schemas/GetStatsConfigResponse'
'DhcpConfig':
'type': 'object'
'properties':
@ -2059,6 +2158,32 @@
'anonymize_client_ip':
'type': 'boolean'
'description': "Anonymize clients' IP addresses"
'GetQueryLogConfigResponse':
'type': 'object'
'description': 'Query log configuration'
'required':
- 'enabled'
- 'interval'
- 'anonymize_client_ip'
- 'ignored'
'properties':
'enabled':
'type': 'boolean'
'description': 'Is query log enabled'
'interval':
'description': >
Time period for query log rotation.
'type': 'number'
'anonymize_client_ip':
'type': 'boolean'
'description': "Anonymize clients' IP addresses"
'ignored':
'description': 'List of host names, which should not be written to log'
'type': 'array'
'items':
'type': 'string'
'PutQueryLogConfigUpdateRequest':
'$ref': '#/components/schemas/GetQueryLogConfigResponse'
'ResultRule':
'description': 'Applied rule.'
'properties':