Generate in CMake header-only library that "stores" its dependency on boost filesystem

I want to generate in cmake a header-only library from header.h, that depends on libboost_system. I can compile the library without problems with:

find_package(Boost COMPONENTS
        system filesystem
        REQUIRED)
include_directories(SYSTEM ${Boost_INCLUDE_DIRS})
add_library(mylib header.h)
target_link_libraries(mylib PUBLIC ${Boost_Libraries})
set_target_properties(mylib PROPERTIES LINKER_LANGUAGE CXX)

But when I link to mylib elsewhere, it fails finding the boost library with a ld error.

The failure makes sense, but I don't know how to solve it in CMake. How do I "store" the boost dependency on mylib? So I don't have to worry about finding the boost library in other external project?

EDIT: I am only having problems testing my library: I have a CMakeLists.txt in a test/ subfolder where my library is generated, and it's there where my test.exe fails to link boost via mylib.

EDIT2: I am using cmake 3.2